Sep 17 14:06:52.964533 ip-10-0-128-175 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 14:06:53.316751 ip-10-0-128-175 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:06:53.316751 ip-10-0-128-175 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 14:06:53.316751 ip-10-0-128-175 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:06:53.316751 ip-10-0-128-175 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 14:06:53.316751 ip-10-0-128-175 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 14:06:53.318094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.318034    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 14:06:53.323125 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323110    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:53.323125 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323125    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323129    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323132    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323135    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323138    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323141    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323144    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323147    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323150    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323154    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323158    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323161    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323164    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323168    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323171    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323175    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323178    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323180    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323183    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:53.323197 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323191    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323194    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323197    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323199    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323202    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323205    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323208    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323210    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323213    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323216    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323219    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323221    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323224    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323226    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323229    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323231    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323234    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323236    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323239    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:53.323671 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323241    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323244    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323246    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323261    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323265    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323267    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323270    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323272    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323275    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323278    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323280    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323283    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323285    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323288    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323296    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323300    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323303    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323306    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323308    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323311    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:53.324167 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323314    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323316    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323319    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323321    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323324    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323327    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323329    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323332    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323334    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323337    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323340    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323342    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323345    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323347    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323349    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323352    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323355    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323358    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323361    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323363    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:53.324668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323366    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323368    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323371    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323373    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323376    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323379    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323382    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323385    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323388    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323391    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323393    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323396    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323398    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323401    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323403    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323406    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323410    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323413    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323416    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:53.325145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.323419    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324079    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324087    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324091    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324094    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324098    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324102    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324105    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324108    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324111    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324113    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324116    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324119    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324121    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324125    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324129    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324131    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324134    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:53.325624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324137    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324139    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324143    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324146    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324149    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324152    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324155    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324157    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324160    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324162    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324165    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324168    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324171    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324173    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324176    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324179    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324181    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324184    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324186    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324189    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324192    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:53.326089 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324194    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324197    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324199    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324203    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324205    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324208    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324210    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324214    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324216    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324219    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324222    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324224    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324227    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324229    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324232    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324235    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324238    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324240    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324242    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324245    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:53.326683 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324248    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324264    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324267    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324270    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324272    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324275    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324277    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324280    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324283    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324285    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324287    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324290    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324293    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324296    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324298    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324301    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324305    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324307    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324310    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324315    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:53.327179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324317    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324320    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324323    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324325    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324328    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324331    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324333    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324336    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324339    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324342    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324345    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324348    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324350    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324352    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324355    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324357    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324360    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324363    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324365    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:53.327698 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324368    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324370    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324700    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324704    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324707    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324710    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324712    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324715    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324717    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324720    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324723    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324725    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324730    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324732    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324737    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324740    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324743    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324746    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324750    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324753    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:53.328179 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324755    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324759    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324761    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324764    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324767    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324770    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324772    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324775    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324777    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324780    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324782    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324786    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324790    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324793    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324795    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324798    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324800    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324803    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324805    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:53.328676 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324809    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324813    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324815    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324818    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324821    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324825    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324828    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324831    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324834    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324836    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324839    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324842    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324845    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324847    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324850    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324853    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324855    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324858    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324860    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324863    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:53.329139 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324865    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324868    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324871    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324873    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324876    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324878    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324881    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324883    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324886    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324888    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324891    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324894    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324896    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324899    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324901    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324904    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324907    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324911    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324913    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324916    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:53.329654 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324918    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324921    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324924    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324926    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324929    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324931    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324934    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324937    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324940    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324942    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324945    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324947    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324950    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324952    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324955    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324957    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324960    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324962    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324965    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324967    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:53.330162 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324969    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.324972    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325040    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325073    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325079    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325083    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325087    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325090    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325094    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325100    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325103    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325105    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325109    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325112    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325115    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325118    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325121    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325124    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325127    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325130    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325133    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325136    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325139    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325143    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 14:06:53.330664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325146    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325150    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325153    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325156    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325159    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325162    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325165    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325168    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325171    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325174    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325177    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325180    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325183    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325186    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325189    2565 flags.go:64] FLAG: --enable-server="true"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325192    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325196    2565 flags.go:64] FLAG: --event-burst="100"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325200    2565 flags.go:64] FLAG: --event-qps="50"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325203    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325206    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325209    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325213    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325215    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325218    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325221    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 17 14:06:53.331287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325224    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325227    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325230    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325233    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325236    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325239    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325242    2565 flags.go:64] FLAG: --feature-gates=""
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325246    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325262    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325265    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325268    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325271    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325274    2565 flags.go:64] FLAG: --help="false"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325277    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325280    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325283    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325285    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325289    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325292    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325295    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325298    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325301    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325304    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 14:06:53.331930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325307    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325312    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325315    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325318    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325320    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325323    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325326    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325331    2565 flags.go:64] FLAG: --lock-file=""
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325334    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325337    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325351    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325365    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325368    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325370    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325373    2565 flags.go:64] FLAG: --logging-format="text"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325377    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325380    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325383    2565 flags.go:64] FLAG: --manifest-url=""
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325386    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325390    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325392    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325396    2565 flags.go:64] FLAG: --max-pods="110"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325399    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325402    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325404    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 14:06:53.332885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325407    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325410    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325413    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325416    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325422    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325425    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325428    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325431    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325435    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325439    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325442    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325445    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325448    2565 flags.go:64] FLAG: --port="10250"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325451    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325456    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c4eb7288e6f52441"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325459    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325462    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325464    2565 flags.go:64] FLAG: --register-node="true"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325467    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325471    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325474    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325477    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325479    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325483    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325486    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 14:06:53.333648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325489    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325492    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325495    2565 flags.go:64] FLAG: --runonce="false"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325497    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325500    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325503    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325506    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325509    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325512    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325515    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325518    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325520    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325523    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325526    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325528    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325531    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325536    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325538    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325544    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325546    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325549    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325554    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325557    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325559    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325562    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 14:06:53.334300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325565    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325568    2565 flags.go:64] FLAG: --v="2"
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325572    2565 flags.go:64] FLAG: --version="false"
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325576    2565 flags.go:64] FLAG: --vmodule=""
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325580    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.325583    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325685    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325689    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325692    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325695    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325698    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325701    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325704    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325707    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325710    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325713    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325716    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325718    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325721    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325725    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325727    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:53.334925 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325731    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325735    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325740    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325743    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325746    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325749    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325752    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325756    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325759    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325762    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325765    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325768    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325771    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325773    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325776    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325779    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325781    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325784    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325787    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325790    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:53.335467 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325792    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325795    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325797    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325800    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325804    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325808    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325811    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325814    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325817    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325819    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325822    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325831    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325834    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325837    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325841    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325843    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325846    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325848    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325851    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:53.335953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325856    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325858    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325861    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325864    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325866    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325869    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325872    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325874    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325878    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325880    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325883    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325885    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325889    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325891    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325894    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325897    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325900    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325903    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325905    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325908    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:53.336474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325910    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325913    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325915    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325918    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325920    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325923    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325925    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325929    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325933    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325936    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325939    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325942    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325946    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325950    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325953    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325956    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325959    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325962    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325965    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325967    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:53.336968 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325970    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325973    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325975    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325978    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.325980    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.326518    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.334656    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.334669    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.334682    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.334690    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334747    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334753    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334758    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334762    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:53.337473 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334766    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334769    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334772    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334775    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334777    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334780    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334783    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334786    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334788    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334791    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334794    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334797    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334799    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334802    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334805    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334808    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334812    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334815    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334818    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334821    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:53.337834 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334823    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334826    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334829    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334831    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334834    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334838    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334841    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334843    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334846    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334848    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334851    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334854    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334857    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334859    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334862    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334865    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334867    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334870    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334872    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334875    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:53.338335 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334878    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334881    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334883    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334886    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334889    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334892    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334894    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334897    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334900    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334903    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334905    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334908    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334910    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334913    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334915    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334918    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334921    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334924    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334927    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:53.338822 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334930    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334933    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334935    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334938    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334941    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334943    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334946    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334949    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334951    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334954    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334957    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334959    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334962    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334964    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334967    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334969    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334980    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334982    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334985    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334988    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:53.339309 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334991    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334993    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334996    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.334998    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335001    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335005    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335009    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335011    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335014    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335017    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335020    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335023    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335026    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335028    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335031    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:53.339851 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335034    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.335039    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335150    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335155    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335158    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335161    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335164    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335167    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335170    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335173    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335175    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335178    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335181    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335183    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335192    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 14:06:53.340225 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335194    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335197    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335199    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335202    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335205    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335208    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335211    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335213    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335216    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335218    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335221    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335224    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335226    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335229    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335232    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335234    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335237    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335239    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335242    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335246    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 14:06:53.340614 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335265    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335270    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335273    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335276    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335279    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335282    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335285    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335287    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335290    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335292    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335295    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335297    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335306    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335309    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335311    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335314    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335317    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335320    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335322    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335325    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 14:06:53.341100 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335327    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335330    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335332    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335335    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335337    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335340    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335342    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335345    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335348    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335351    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335353    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335356    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335358    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335361    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335364    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335366    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335368    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335371    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335373    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335376    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 14:06:53.341668 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335378    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335382    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335385    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335388    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335390    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335398    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335402    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335405    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335408    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335411    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335414    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335417    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335419    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335422    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335424    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335427    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335430    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335432    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 14:06:53.342151 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335435    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335438    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335440    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335443    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335445    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335448    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335450    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:53.335453    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.335457    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.335542    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.339332    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.340152    2565 server.go:1019] "Starting client certificate rotation"
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.340246    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 14:06:53.342624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.340297    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 14:06:53.342952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.342486    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 14:06:53.346420 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.346404    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 14:06:53.354418 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.354404    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 17 14:06:53.357080 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.357065    2565 log.go:25] "Validated CRI v1 image API"
Sep 17 14:06:53.357947 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.357929    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 14:06:53.361702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.361684    2565 fs.go:135] Filesystem UUIDs: map[1db69f42-f9ec-4818-a980-2abff2e3e98b:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c5f5bc46-7d14-4aa2-831c-9edb35692d40:/dev/nvme0n1p3]
Sep 17 14:06:53.361758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.361702    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 14:06:53.367182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.367084    2565 manager.go:216] Machine: {Timestamp:2026-09-17 14:06:53.365922542 +0000 UTC m=+0.299427725 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095041 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2053ab1f689cbb0a00089bf82e2eec SystemUUID:ec2053ab-1f68-9cbb-0a00-089bf82e2eec BootID:8a822607-1799-4004-9b16-362fa21ef9a5 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e9:4a:d0:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e9:4a:d0:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:1b:be:05:75:22 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 14:06:53.367182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.367178    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 14:06:53.367297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.367238    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 14:06:53.371661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.371635    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 14:06:53.371784 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.371663    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-175.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 14:06:53.372878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.372868    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 14:06:53.372932 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.372880    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 14:06:53.372932 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.372887    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 14:06:53.373434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.373424    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 14:06:53.374577 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.374566    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 14:06:53.378339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.378327    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 14:06:53.378445 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.378435    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 14:06:53.378490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.378466    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 14:06:53.378490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.378475    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 14:06:53.378490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.378488    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 14:06:53.378606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.378497    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 17 14:06:53.378606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.378506    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 14:06:53.379674 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.379657    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 14:06:53.379749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.379687    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 14:06:53.382572 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.382548    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 17 14:06:53.384493 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.384478    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 14:06:53.384564 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.384509    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 14:06:53.386029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386018    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386034    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386040    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386044    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386052    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386057    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386070    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386075    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386081    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 14:06:53.386083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386086    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 14:06:53.386377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386094    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 14:06:53.386377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.386100    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 14:06:53.387757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.387743    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 14:06:53.387757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.387755    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 14:06:53.391028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.391014    2565 server.go:1268] "Started kubelet"
Sep 17 14:06:53.391149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.391099    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 14:06:53.391210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.391169    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 14:06:53.391288 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.391121    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 14:06:53.391798 ip-10-0-128-175 systemd[1]: Started Kubernetes Kubelet.
Sep 17 14:06:53.392680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.392663    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 14:06:53.393385 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.393368    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 14:06:53.393889 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.393863    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 14:06:53.397113 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.397094    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 14:06:53.397207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.397102    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 14:06:53.397282 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.397220    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 14:06:53.397336 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.397315    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:53.397883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.397865    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 14:06:53.397975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.397913    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 14:06:53.397975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.397953    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 14:06:53.398076 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398064    2565 factory.go:55] Registering systemd factory
Sep 17 14:06:53.398076 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398071    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 14:06:53.398212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398079    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 17 14:06:53.398212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398081    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 14:06:53.398326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398308    2565 factory.go:153] Registering CRI-O factory
Sep 17 14:06:53.398378 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398335    2565 factory.go:223] Registration of the crio container factory successfully
Sep 17 14:06:53.398426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398386    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 14:06:53.398426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398403    2565 factory.go:103] Registering Raw factory
Sep 17 14:06:53.398525 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398441    2565 manager.go:1196] Started watching for new ooms in manager
Sep 17 14:06:53.398864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.398795    2565 manager.go:349] Starting recovery of all containers
Sep 17 14:06:53.401353 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.401329    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 14:06:53.401599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.401585    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-175.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 14:06:53.401654 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.401641    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 14:06:53.401687 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.401657    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 14:06:53.401742 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.401728    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 14:06:53.402191 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.401381    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-175.ec2.internal.18d620d7db05a675  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-175.ec2.internal,UID:ip-10-0-128-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-175.ec2.internal,},FirstTimestamp:2026-09-17 14:06:53.390988917 +0000 UTC m=+0.324494100,LastTimestamp:2026-09-17 14:06:53.390988917 +0000 UTC m=+0.324494100,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-175.ec2.internal,}"
Sep 17 14:06:53.410057 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.409922    2565 manager.go:354] Recovery completed
Sep 17 14:06:53.414731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.414639    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:53.416170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.416143    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:53.416234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.416182    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:53.416234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.416196    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:53.416769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.416754    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 14:06:53.416769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.416766    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 14:06:53.416875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.416780    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 14:06:53.416875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.416792    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:53.417529 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.417511    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:53.417629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.417542    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:53.417629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.417555    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:53.418541 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.418464    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-175.ec2.internal.18d620d7dc85e075  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-175.ec2.internal,UID:ip-10-0-128-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-175.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-175.ec2.internal,},FirstTimestamp:2026-09-17 14:06:53.416169589 +0000 UTC m=+0.349674775,LastTimestamp:2026-09-17 14:06:53.416169589 +0000 UTC m=+0.349674775,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-175.ec2.internal,}"
Sep 17 14:06:53.418969 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.418956    2565 policy_none.go:49] "None policy: Start"
Sep 17 14:06:53.419033 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.418972    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 14:06:53.419033 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.418985    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 14:06:53.420943 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.420928    2565 policy_none.go:47] "Start"
Sep 17 14:06:53.433237 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.433221    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chrbt"
Sep 17 14:06:53.436218 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.436136    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-175.ec2.internal.18d620d7dc86297d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-175.ec2.internal,UID:ip-10-0-128-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-175.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-175.ec2.internal,},FirstTimestamp:2026-09-17 14:06:53.416188285 +0000 UTC m=+0.349693469,LastTimestamp:2026-09-17 14:06:53.416188285 +0000 UTC m=+0.349693469,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-175.ec2.internal,}"
Sep 17 14:06:53.439376 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.439315    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-175.ec2.internal.18d620d7dc865d05  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-175.ec2.internal,UID:ip-10-0-128-175.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-175.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-175.ec2.internal,},FirstTimestamp:2026-09-17 14:06:53.416201477 +0000 UTC m=+0.349706662,LastTimestamp:2026-09-17 14:06:53.416201477 +0000 UTC m=+0.349706662,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-175.ec2.internal,}"
Sep 17 14:06:53.447836 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.447821    2565 manager.go:337] "Starting Device Plugin manager"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.447865    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.447877    2565 server.go:87] "Starting device plugin registration server"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.448132    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.448143    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.448279    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.448346    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.448355    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.448842    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.448875    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:53.473346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.450995    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-chrbt"
Sep 17 14:06:53.511685 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.511651    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 14:06:53.512818 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.512804    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 14:06:53.512874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.512844    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 14:06:53.512874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.512864    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 14:06:53.512957 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.512900    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 14:06:53.517602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.517585    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:53.549165 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.549130    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:53.550826 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.550813    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:53.550900 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.550839    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:53.550900 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.550854    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:53.550900 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.550878    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.565085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.565066    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.565152 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.565086    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-175.ec2.internal\": node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:53.612576 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.612523    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:53.613566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.613548    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal"]
Sep 17 14:06:53.613633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.613621    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:53.614929 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.614916    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:53.615011 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.614944    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:53.615011 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.614956    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:53.617038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617024    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:53.617151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.617193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617166    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:53.617686 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617669    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:53.617757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617695    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:53.617757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617705    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:53.617757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617671    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:53.617908 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617778    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:53.617908 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.617794    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:53.619701 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.619687    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.619750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.619713    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 14:06:53.620356 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.620340    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 14:06:53.620446 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.620362    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 14:06:53.620446 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.620375    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeHasSufficientPID"
Sep 17 14:06:53.652795 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.652778    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-175.ec2.internal\" not found" node="ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.657065 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.657049    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-175.ec2.internal\" not found" node="ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.700386 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.700365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82c31071652d7cafc3e9e053617363c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal\" (UID: \"82c31071652d7cafc3e9e053617363c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.700472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.700392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82c31071652d7cafc3e9e053617363c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal\" (UID: \"82c31071652d7cafc3e9e053617363c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.700472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.700424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fd9853fb4ec76c374d0d4c456a7e8b2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.ec2.internal\" (UID: \"4fd9853fb4ec76c374d0d4c456a7e8b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.713488 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.713474    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:53.801496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.801475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82c31071652d7cafc3e9e053617363c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal\" (UID: \"82c31071652d7cafc3e9e053617363c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.801559 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.801504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82c31071652d7cafc3e9e053617363c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal\" (UID: \"82c31071652d7cafc3e9e053617363c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.801559 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.801523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/82c31071652d7cafc3e9e053617363c6-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal\" (UID: \"82c31071652d7cafc3e9e053617363c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.801559 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.801520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fd9853fb4ec76c374d0d4c456a7e8b2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.ec2.internal\" (UID: \"4fd9853fb4ec76c374d0d4c456a7e8b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.801653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.801571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/82c31071652d7cafc3e9e053617363c6-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal\" (UID: \"82c31071652d7cafc3e9e053617363c6\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.801653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.801603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4fd9853fb4ec76c374d0d4c456a7e8b2-config\") pod \"kube-apiserver-proxy-ip-10-0-128-175.ec2.internal\" (UID: \"4fd9853fb4ec76c374d0d4c456a7e8b2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.813546 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.813518    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:53.914328 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:53.914278    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:53.954896 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.954875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:53.960403 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:53.960325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:54.014557 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.014532    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:54.114972 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.114947    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:54.215466 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.215422    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-175.ec2.internal\" not found"
Sep 17 14:06:54.287739 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.287716    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:54.298614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.298590    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:54.339793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.339770    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 14:06:54.340145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.339882    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 14:06:54.340145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.339945    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 14:06:54.340145 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.339987    2565 kubelet.go:3250] "Failed creating a mirror pod" err="Post \"https://a076f929e6e26432680cee0b22388f88-42d64be0a4f07f51.elb.us-east-1.amazonaws.com:6443/api/v1/namespaces/openshift-machine-config-operator/pods\": read tcp 10.0.128.175:38986->32.199.20.155:6443: use of closed network connection" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:54.340145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.340029    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal"
Sep 17 14:06:54.378849 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.378826    2565 apiserver.go:52] "Watching apiserver"
Sep 17 14:06:54.394063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.394043    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 14:06:54.394491 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.394471    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-8f6tm","kube-system/global-pull-secret-syncer-52tc9","kube-system/konnectivity-agent-46lvw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal","openshift-multus/multus-additional-cni-plugins-7vlwh","openshift-multus/network-metrics-daemon-h5mnt","openshift-network-diagnostics/network-check-target-x8285","openshift-cluster-node-tuning-operator/tuned-ggczn","openshift-dns/node-resolver-dxm97","openshift-image-registry/node-ca-hztvs","openshift-multus/multus-gt5cj","openshift-network-operator/iptables-alerter-qrls6"]
Sep 17 14:06:54.397146 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.397127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:54.397235 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.397211    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:06:54.398053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.398036    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 14:06:54.398648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.398629    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 14:06:54.399343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.399319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.399467 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.399389    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:06:54.401535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.401516    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:06:54.403825 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.403805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.405546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.405527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3a2822e-00c7-4ac7-85cc-816306696a1f-dbus\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.405649 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.405554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.405649 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.405577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:54.405649 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.405598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlwbb\" (UniqueName: \"kubernetes.io/projected/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-kube-api-access-hlwbb\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:54.405649 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.405621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3a2822e-00c7-4ac7-85cc-816306696a1f-kubelet-config\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.406249 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.406231    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.408226 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.408201    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 14:06:54.408356 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.408316    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bb8x9\""
Sep 17 14:06:54.408739 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.408721    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.411190 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.411170    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:54.411287 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.411240    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:06:54.413809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.413555    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-74rlx\""
Sep 17 14:06:54.413809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.413581    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 14:06:54.413809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.413592    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 14:06:54.413809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.413605    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 14:06:54.413809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.413748    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 14:06:54.415312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.414259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.415312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.414403    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 14:06:54.415312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.414607    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 14:06:54.415312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.414619    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pmzkn\""
Sep 17 14:06:54.415312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.414866    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 14:06:54.415312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.414937    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 14:06:54.415312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.415113    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 14:06:54.415312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.415208    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 14:06:54.415771 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.415426    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 14:06:54.415771 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.415427    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 14:06:54.416059 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.416040    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 14:06:54.416416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.416382    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 14:06:54.417602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.416591    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 14:06:54.417602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.416674    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.418457 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.418056    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7zh6m\""
Sep 17 14:06:54.420164 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.420143    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.422615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.422595    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.424959 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.424942    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal"]
Sep 17 14:06:54.425030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.425015    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.430867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.430851    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5tcgh\""
Sep 17 14:06:54.431394 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.431381    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 14:06:54.431675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.431662    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 14:06:54.433271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.433249    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:06:54.433868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.433841    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 14:06:54.434055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.434025    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 14:06:54.436287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.436252    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 14:06:54.436470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.436285    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-bnkqp\""
Sep 17 14:06:54.436584 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.436567    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-px4kj\""
Sep 17 14:06:54.436957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.436941    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 14:06:54.437253 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.437239    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-bnqtd\""
Sep 17 14:06:54.437983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.437968    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 14:06:54.438794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.438770    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 14:06:54.439666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.439650    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 14:06:54.439930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.439916    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftmvf\""
Sep 17 14:06:54.440834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.440270    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 14:06:54.440834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.440669    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:06:54.454368 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.454296    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 14:01:54 +0000 UTC" deadline="2028-02-26 10:43:01.89015918 +0000 UTC"
Sep 17 14:06:54.454368 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.454367    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12644h36m7.435795324s"
Sep 17 14:06:54.479171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.479124    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttqtg"
Sep 17 14:06:54.499869 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.499851    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 14:06:54.500030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.500011    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-ttqtg"
Sep 17 14:06:54.506030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506011    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3a2822e-00c7-4ac7-85cc-816306696a1f-kubelet-config\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.506118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.506118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-systemd\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.506118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506084    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-etc-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.506118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-socket-dir-parent\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.506296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f87fcc-0335-4b5d-b1c0-65a17867de35-host-slash\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.506296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c3a2822e-00c7-4ac7-85cc-816306696a1f-kubelet-config\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.506296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:54.506296 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.506210    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:54.506296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506235    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-677ct\" (UniqueName: \"kubernetes.io/projected/c41caec2-34bf-49de-a880-3ca09c3ed778-kube-api-access-677ct\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.506523 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.506304    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret podName:c3a2822e-00c7-4ac7-85cc-816306696a1f nodeName:}" failed. No retries permitted until 2026-09-17 14:06:55.006270447 +0000 UTC m=+1.939775638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret") pod "global-pull-secret-syncer-52tc9" (UID: "c3a2822e-00c7-4ac7-85cc-816306696a1f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:54.506523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506320    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-sys-fs\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.506523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p49xt\" (UniqueName: \"kubernetes.io/projected/f2312773-bed0-40d9-a40f-1b8f822d4c98-kube-api-access-p49xt\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.506523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506366    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-system-cni-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.506523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:54.506523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.506523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-kubelet\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.506525    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-ovnkube-script-lib\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.506567    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs podName:e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d nodeName:}" failed. No retries permitted until 2026-09-17 14:06:55.006556324 +0000 UTC m=+1.940061502 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs") pod "network-metrics-daemon-h5mnt" (UID: "e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506584    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-registration-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-kubelet\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3a2822e-00c7-4ac7-85cc-816306696a1f-dbus\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cnibin\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506698    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-var-lib-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-systemd\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c3a2822e-00c7-4ac7-85cc-816306696a1f-dbus\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-etc-kubernetes\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c3a0f5c-65b9-46f6-a43d-08b7b7427fef-agent-certs\") pod \"konnectivity-agent-46lvw\" (UID: \"8c3a0f5c-65b9-46f6-a43d-08b7b7427fef\") " pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:06:54.506779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-run-netns\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-ovn\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-log-socket\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506839    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-modprobe-d\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-socket-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-hosts-file\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-cni-bin\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.506972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c41caec2-34bf-49de-a880-3ca09c3ed778-tmp\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-conf-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507046    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxzx\" (UniqueName: \"kubernetes.io/projected/21f87fcc-0335-4b5d-b1c0-65a17867de35-kube-api-access-2vxzx\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-cni-bin\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-tuned\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.507522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-cnibin\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-daemon-config\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-run\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-system-cni-dir\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507332    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-systemd-units\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc4jv\" (UniqueName: \"kubernetes.io/projected/66d0451d-5e95-4e84-bf53-0df798ea36df-kube-api-access-xc4jv\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-netns\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507405    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-hostroot\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-cni-netd\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507485    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7wxf\" (UniqueName: \"kubernetes.io/projected/a6762df1-b224-4466-ad96-74ee1af45c5f-kube-api-access-z7wxf\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysctl-d\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysctl-conf\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-cni-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-ovnkube-config\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysconfig\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.508234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-tmp-dir\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f87fcc-0335-4b5d-b1c0-65a17867de35-iptables-alerter-script\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-env-overrides\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-kubernetes\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-sys\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-lib-modules\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507803    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m726q\" (UniqueName: \"kubernetes.io/projected/260ac517-f5f9-421e-b392-3155fa34f1ae-kube-api-access-m726q\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlwbb\" (UniqueName: \"kubernetes.io/projected/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-kube-api-access-hlwbb\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507869    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlr2w\" (UniqueName: \"kubernetes.io/projected/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-kube-api-access-xlr2w\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507897    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2312773-bed0-40d9-a40f-1b8f822d4c98-serviceca\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrn74\" (UniqueName: \"kubernetes.io/projected/886834cb-4fbd-4b7d-a883-0aa36121b1d6-kube-api-access-nrn74\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507952    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6762df1-b224-4466-ad96-74ee1af45c5f-ovn-node-metrics-cert\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.507975    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508005    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2312773-bed0-40d9-a40f-1b8f822d4c98-host\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-cni-multus\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.508791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a0f5c-65b9-46f6-a43d-08b7b7427fef-konnectivity-ca\") pod \"konnectivity-agent-46lvw\" (UID: \"8c3a0f5c-65b9-46f6-a43d-08b7b7427fef\") " pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:06:54.509267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-slash\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.509267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508124    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-node-log\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.509267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508148    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-var-lib-kubelet\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.509267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-host\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.509267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508216    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-device-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.509267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508236    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/260ac517-f5f9-421e-b392-3155fa34f1ae-cni-binary-copy\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.509267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.508289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-multus-certs\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.538229 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.538204    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 14:06:54.541598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.541577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlwbb\" (UniqueName: \"kubernetes.io/projected/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-kube-api-access-hlwbb\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:54.556171 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.556136    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c31071652d7cafc3e9e053617363c6.slice/crio-ecb4d1ed79f2a352c3ea5bf5fcf0621b22eb7169380e899eda60f3047696a810 WatchSource:0}: Error finding container ecb4d1ed79f2a352c3ea5bf5fcf0621b22eb7169380e899eda60f3047696a810: Status 404 returned error can't find the container with id ecb4d1ed79f2a352c3ea5bf5fcf0621b22eb7169380e899eda60f3047696a810
Sep 17 14:06:54.556494 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.556475    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd9853fb4ec76c374d0d4c456a7e8b2.slice/crio-0e2d95f42b9b508ba1682974f5987eb469da21aca3e1488693868a9912271886 WatchSource:0}: Error finding container 0e2d95f42b9b508ba1682974f5987eb469da21aca3e1488693868a9912271886: Status 404 returned error can't find the container with id 0e2d95f42b9b508ba1682974f5987eb469da21aca3e1488693868a9912271886
Sep 17 14:06:54.562765 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.562750    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:06:54.608553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-ovnkube-config\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.608658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608558    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysconfig\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.608658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-tmp-dir\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.608658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f87fcc-0335-4b5d-b1c0-65a17867de35-iptables-alerter-script\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.608658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.608658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-env-overrides\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-kubernetes\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608669    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysconfig\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-sys\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-lib-modules\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m726q\" (UniqueName: \"kubernetes.io/projected/260ac517-f5f9-421e-b392-3155fa34f1ae-kube-api-access-m726q\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlr2w\" (UniqueName: \"kubernetes.io/projected/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-kube-api-access-xlr2w\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-sys\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608779    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2312773-bed0-40d9-a40f-1b8f822d4c98-serviceca\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrn74\" (UniqueName: \"kubernetes.io/projected/886834cb-4fbd-4b7d-a883-0aa36121b1d6-kube-api-access-nrn74\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608867    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6762df1-b224-4466-ad96-74ee1af45c5f-ovn-node-metrics-cert\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.608927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608920    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2312773-bed0-40d9-a40f-1b8f822d4c98-host\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-cni-multus\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a0f5c-65b9-46f6-a43d-08b7b7427fef-konnectivity-ca\") pod \"konnectivity-agent-46lvw\" (UID: \"8c3a0f5c-65b9-46f6-a43d-08b7b7427fef\") " pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-slash\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-node-log\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-var-lib-kubelet\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-host\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609073    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-host\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609117    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-lib-modules\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-kubernetes\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609204    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f2312773-bed0-40d9-a40f-1b8f822d4c98-serviceca\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609252    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/21f87fcc-0335-4b5d-b1c0-65a17867de35-iptables-alerter-script\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609226    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-device-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-ovnkube-config\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/260ac517-f5f9-421e-b392-3155fa34f1ae-cni-binary-copy\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-slash\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609340    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-cni-multus\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-node-log\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.609593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-multus-certs\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609379    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-var-lib-kubelet\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2312773-bed0-40d9-a40f-1b8f822d4c98-host\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-systemd\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-device-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609470    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-etc-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609498    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-socket-dir-parent\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.608988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-tmp-dir\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f87fcc-0335-4b5d-b1c0-65a17867de35-host-slash\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609557    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-677ct\" (UniqueName: \"kubernetes.io/projected/c41caec2-34bf-49de-a880-3ca09c3ed778-kube-api-access-677ct\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-multus-certs\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-etc-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-sys-fs\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-socket-dir-parent\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.610395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p49xt\" (UniqueName: \"kubernetes.io/projected/f2312773-bed0-40d9-a40f-1b8f822d4c98-kube-api-access-p49xt\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-systemd\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-system-cni-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609726    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-env-overrides\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/21f87fcc-0335-4b5d-b1c0-65a17867de35-host-slash\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-system-cni-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-sys-fs\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609923    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-kubelet\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609977    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/260ac517-f5f9-421e-b392-3155fa34f1ae-cni-binary-copy\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.609990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-kubelet\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8c3a0f5c-65b9-46f6-a43d-08b7b7427fef-konnectivity-ca\") pod \"konnectivity-agent-46lvw\" (UID: \"8c3a0f5c-65b9-46f6-a43d-08b7b7427fef\") " pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610034    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-ovnkube-script-lib\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-run-ovn-kubernetes\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-registration-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.610983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-registration-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-kubelet\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cnibin\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-var-lib-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610297    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cnibin\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610315    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-systemd\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-etc-kubernetes\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-var-lib-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c3a0f5c-65b9-46f6-a43d-08b7b7427fef-agent-certs\") pod \"konnectivity-agent-46lvw\" (UID: \"8c3a0f5c-65b9-46f6-a43d-08b7b7427fef\") " pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610388    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-systemd\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-run-netns\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610422    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-kubelet\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610456    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-run-netns\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-ovn\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-etc-kubernetes\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-log-socket\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.611893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610531    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-ovn\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-modprobe-d\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610580    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a6762df1-b224-4466-ad96-74ee1af45c5f-ovnkube-script-lib\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-socket-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610634    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-log-socket\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610680    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-modprobe-d\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610706    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-hosts-file\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/66d0451d-5e95-4e84-bf53-0df798ea36df-socket-dir\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-hosts-file\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610762    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-k8s-cni-cncf-io\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-cni-bin\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-var-lib-cni-bin\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c41caec2-34bf-49de-a880-3ca09c3ed778-tmp\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-conf-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxzx\" (UniqueName: \"kubernetes.io/projected/21f87fcc-0335-4b5d-b1c0-65a17867de35-kube-api-access-2vxzx\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.612842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610908    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-cni-bin\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.610956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-tuned\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611022    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611024    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-conf-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611057    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-run-openvswitch\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-cnibin\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-daemon-config\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-run\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-system-cni-dir\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611176    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-systemd-units\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc4jv\" (UniqueName: \"kubernetes.io/projected/66d0451d-5e95-4e84-bf53-0df798ea36df-kube-api-access-xc4jv\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-cni-binary-copy\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-netns\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611356    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-host-run-netns\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-hostroot\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.613598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611403    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-systemd-units\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611405    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/886834cb-4fbd-4b7d-a883-0aa36121b1d6-system-cni-dir\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611412    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-cni-netd\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-run\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611471    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7wxf\" (UniqueName: \"kubernetes.io/projected/a6762df1-b224-4466-ad96-74ee1af45c5f-kube-api-access-z7wxf\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysctl-d\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-cnibin\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysctl-conf\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-cni-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-cni-netd\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611651    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-daemon-config\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-multus-cni-dir\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611671    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysctl-d\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/260ac517-f5f9-421e-b392-3155fa34f1ae-hostroot\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-sysctl-conf\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.611837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6762df1-b224-4466-ad96-74ee1af45c5f-host-cni-bin\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.612173    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a6762df1-b224-4466-ad96-74ee1af45c5f-ovn-node-metrics-cert\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.614088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.612768    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/886834cb-4fbd-4b7d-a883-0aa36121b1d6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.614680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.613062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8c3a0f5c-65b9-46f6-a43d-08b7b7427fef-agent-certs\") pod \"konnectivity-agent-46lvw\" (UID: \"8c3a0f5c-65b9-46f6-a43d-08b7b7427fef\") " pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:06:54.614680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.613711    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c41caec2-34bf-49de-a880-3ca09c3ed778-etc-tuned\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.614680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.614036    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c41caec2-34bf-49de-a880-3ca09c3ed778-tmp\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.642768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.642748    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m726q\" (UniqueName: \"kubernetes.io/projected/260ac517-f5f9-421e-b392-3155fa34f1ae-kube-api-access-m726q\") pod \"multus-gt5cj\" (UID: \"260ac517-f5f9-421e-b392-3155fa34f1ae\") " pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.644922 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.644895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrn74\" (UniqueName: \"kubernetes.io/projected/886834cb-4fbd-4b7d-a883-0aa36121b1d6-kube-api-access-nrn74\") pod \"multus-additional-cni-plugins-7vlwh\" (UID: \"886834cb-4fbd-4b7d-a883-0aa36121b1d6\") " pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.649293 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.649272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxzx\" (UniqueName: \"kubernetes.io/projected/21f87fcc-0335-4b5d-b1c0-65a17867de35-kube-api-access-2vxzx\") pod \"iptables-alerter-qrls6\" (UID: \"21f87fcc-0335-4b5d-b1c0-65a17867de35\") " pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.649429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.649411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-677ct\" (UniqueName: \"kubernetes.io/projected/c41caec2-34bf-49de-a880-3ca09c3ed778-kube-api-access-677ct\") pod \"tuned-ggczn\" (UID: \"c41caec2-34bf-49de-a880-3ca09c3ed778\") " pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.650534 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.650517    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:54.650580 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.650542    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:54.650580 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.650557    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wlrr4 for pod openshift-network-diagnostics/network-check-target-x8285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:54.650679 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:54.650619    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4 podName:5af832b9-d1bc-4c75-8703-dc0dfdd6b174 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:55.150599519 +0000 UTC m=+2.084104715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wlrr4" (UniqueName: "kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4") pod "network-check-target-x8285" (UID: "5af832b9-d1bc-4c75-8703-dc0dfdd6b174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:54.650934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.650913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlr2w\" (UniqueName: \"kubernetes.io/projected/4d20f337-ad17-4d2c-a2d3-e4e7c15b162d-kube-api-access-xlr2w\") pod \"node-resolver-dxm97\" (UID: \"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d\") " pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.652530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.652510    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc4jv\" (UniqueName: \"kubernetes.io/projected/66d0451d-5e95-4e84-bf53-0df798ea36df-kube-api-access-xc4jv\") pod \"aws-ebs-csi-driver-node-4fwk5\" (UID: \"66d0451d-5e95-4e84-bf53-0df798ea36df\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.653053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.653037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7wxf\" (UniqueName: \"kubernetes.io/projected/a6762df1-b224-4466-ad96-74ee1af45c5f-kube-api-access-z7wxf\") pod \"ovnkube-node-8f6tm\" (UID: \"a6762df1-b224-4466-ad96-74ee1af45c5f\") " pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.654344 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.654314    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p49xt\" (UniqueName: \"kubernetes.io/projected/f2312773-bed0-40d9-a40f-1b8f822d4c98-kube-api-access-p49xt\") pod \"node-ca-hztvs\" (UID: \"f2312773-bed0-40d9-a40f-1b8f822d4c98\") " pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.736846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.736798    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:06:54.747064 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.747042    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3a0f5c_65b9_46f6_a43d_08b7b7427fef.slice/crio-2c46a53cc41d57061172a066eb1702c38bfeddda15b60cec1482704bef2ba3bf WatchSource:0}: Error finding container 2c46a53cc41d57061172a066eb1702c38bfeddda15b60cec1482704bef2ba3bf: Status 404 returned error can't find the container with id 2c46a53cc41d57061172a066eb1702c38bfeddda15b60cec1482704bef2ba3bf
Sep 17 14:06:54.747064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.747058    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5"
Sep 17 14:06:54.754977 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.754958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7vlwh"
Sep 17 14:06:54.755248 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.755226    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d0451d_5e95_4e84_bf53_0df798ea36df.slice/crio-90e8172b7a33134a45cc09f3438e10d9a0339edf9759c5fd1977f9b8115983d1 WatchSource:0}: Error finding container 90e8172b7a33134a45cc09f3438e10d9a0339edf9759c5fd1977f9b8115983d1: Status 404 returned error can't find the container with id 90e8172b7a33134a45cc09f3438e10d9a0339edf9759c5fd1977f9b8115983d1
Sep 17 14:06:54.765620 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.764614    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886834cb_4fbd_4b7d_a883_0aa36121b1d6.slice/crio-9ae369fc0f5c90d5168243dcd8469b4a24db3363c449cca6e64dcbb8f30aa7b4 WatchSource:0}: Error finding container 9ae369fc0f5c90d5168243dcd8469b4a24db3363c449cca6e64dcbb8f30aa7b4: Status 404 returned error can't find the container with id 9ae369fc0f5c90d5168243dcd8469b4a24db3363c449cca6e64dcbb8f30aa7b4
Sep 17 14:06:54.779304 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.779283    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:06:54.787294 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.787276    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6762df1_b224_4466_ad96_74ee1af45c5f.slice/crio-c38b32708da242002cd9155b37b495baf82b57fb9a0a965bd2c5f7ae538d9922 WatchSource:0}: Error finding container c38b32708da242002cd9155b37b495baf82b57fb9a0a965bd2c5f7ae538d9922: Status 404 returned error can't find the container with id c38b32708da242002cd9155b37b495baf82b57fb9a0a965bd2c5f7ae538d9922
Sep 17 14:06:54.799738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.799719    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ggczn"
Sep 17 14:06:54.807805 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.807661    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41caec2_34bf_49de_a880_3ca09c3ed778.slice/crio-e1092218f765999a454ffe2bb3df56b52c5df704603e2371a3f78b1b78f9c617 WatchSource:0}: Error finding container e1092218f765999a454ffe2bb3df56b52c5df704603e2371a3f78b1b78f9c617: Status 404 returned error can't find the container with id e1092218f765999a454ffe2bb3df56b52c5df704603e2371a3f78b1b78f9c617
Sep 17 14:06:54.817193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.817175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dxm97"
Sep 17 14:06:54.825007 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.824985    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d20f337_ad17_4d2c_a2d3_e4e7c15b162d.slice/crio-c5e411e813e4cd59ce4a223ecc6462ead21bdc76406b0c8e3e6c65f0063dc946 WatchSource:0}: Error finding container c5e411e813e4cd59ce4a223ecc6462ead21bdc76406b0c8e3e6c65f0063dc946: Status 404 returned error can't find the container with id c5e411e813e4cd59ce4a223ecc6462ead21bdc76406b0c8e3e6c65f0063dc946
Sep 17 14:06:54.838289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.838270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hztvs"
Sep 17 14:06:54.846231 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.846209    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2312773_bed0_40d9_a40f_1b8f822d4c98.slice/crio-1213669ce20002451fd6f53375ea7e3dcd1fcd36f498116f80f21a32821cb69e WatchSource:0}: Error finding container 1213669ce20002451fd6f53375ea7e3dcd1fcd36f498116f80f21a32821cb69e: Status 404 returned error can't find the container with id 1213669ce20002451fd6f53375ea7e3dcd1fcd36f498116f80f21a32821cb69e
Sep 17 14:06:54.851232 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.851202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gt5cj"
Sep 17 14:06:54.860402 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.860381    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260ac517_f5f9_421e_b392_3155fa34f1ae.slice/crio-4f2650ee5dc7476853cfa7dbf3a49e35299cc2aac1eb1aa62c803c9478dc16f3 WatchSource:0}: Error finding container 4f2650ee5dc7476853cfa7dbf3a49e35299cc2aac1eb1aa62c803c9478dc16f3: Status 404 returned error can't find the container with id 4f2650ee5dc7476853cfa7dbf3a49e35299cc2aac1eb1aa62c803c9478dc16f3
Sep 17 14:06:54.861400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.861382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qrls6"
Sep 17 14:06:54.869527 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:06:54.869506    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21f87fcc_0335_4b5d_b1c0_65a17867de35.slice/crio-d75057dd50a0c12d640a9fc82251edce65d8503406b547dab0864eefd7e210d2 WatchSource:0}: Error finding container d75057dd50a0c12d640a9fc82251edce65d8503406b547dab0864eefd7e210d2: Status 404 returned error can't find the container with id d75057dd50a0c12d640a9fc82251edce65d8503406b547dab0864eefd7e210d2
Sep 17 14:06:54.869805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.869789    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:54.918835 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:54.918813    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:55.014665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.014602    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:55.014665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.014656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:55.014822 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.014743    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:55.014822 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.014758    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:55.014939 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.014824    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret podName:c3a2822e-00c7-4ac7-85cc-816306696a1f nodeName:}" failed. No retries permitted until 2026-09-17 14:06:56.014802396 +0000 UTC m=+2.948307583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret") pod "global-pull-secret-syncer-52tc9" (UID: "c3a2822e-00c7-4ac7-85cc-816306696a1f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:55.014939 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.014844    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs podName:e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d nodeName:}" failed. No retries permitted until 2026-09-17 14:06:56.014833992 +0000 UTC m=+2.948339167 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs") pod "network-metrics-daemon-h5mnt" (UID: "e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:55.211899 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.211872    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:55.216194 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.216165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:55.216413 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.216392    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:55.216491 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.216423    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:55.216491 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.216437    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wlrr4 for pod openshift-network-diagnostics/network-check-target-x8285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:55.216602 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.216496    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4 podName:5af832b9-d1bc-4c75-8703-dc0dfdd6b174 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:56.216477637 +0000 UTC m=+3.149982826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlrr4" (UniqueName: "kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4") pod "network-check-target-x8285" (UID: "5af832b9-d1bc-4c75-8703-dc0dfdd6b174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:55.503450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.503337    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 14:01:55 +0000 UTC" deadline="2028-05-31 04:07:58.961718214 +0000 UTC"
Sep 17 14:06:55.503450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.503402    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14918h1m3.45832118s"
Sep 17 14:06:55.514475 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.514424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:55.514608 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:55.514558    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:06:55.525706 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.525655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" event={"ID":"886834cb-4fbd-4b7d-a883-0aa36121b1d6","Type":"ContainerStarted","Data":"9ae369fc0f5c90d5168243dcd8469b4a24db3363c449cca6e64dcbb8f30aa7b4"}
Sep 17 14:06:55.530860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.530834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5" event={"ID":"66d0451d-5e95-4e84-bf53-0df798ea36df","Type":"ContainerStarted","Data":"90e8172b7a33134a45cc09f3438e10d9a0339edf9759c5fd1977f9b8115983d1"}
Sep 17 14:06:55.543201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.543169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal" event={"ID":"4fd9853fb4ec76c374d0d4c456a7e8b2","Type":"ContainerStarted","Data":"0e2d95f42b9b508ba1682974f5987eb469da21aca3e1488693868a9912271886"}
Sep 17 14:06:55.570402 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.570299    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal" event={"ID":"82c31071652d7cafc3e9e053617363c6","Type":"ContainerStarted","Data":"ecb4d1ed79f2a352c3ea5bf5fcf0621b22eb7169380e899eda60f3047696a810"}
Sep 17 14:06:55.595406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.595341    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qrls6" event={"ID":"21f87fcc-0335-4b5d-b1c0-65a17867de35","Type":"ContainerStarted","Data":"d75057dd50a0c12d640a9fc82251edce65d8503406b547dab0864eefd7e210d2"}
Sep 17 14:06:55.600411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.600383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hztvs" event={"ID":"f2312773-bed0-40d9-a40f-1b8f822d4c98","Type":"ContainerStarted","Data":"1213669ce20002451fd6f53375ea7e3dcd1fcd36f498116f80f21a32821cb69e"}
Sep 17 14:06:55.611509 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.611479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggczn" event={"ID":"c41caec2-34bf-49de-a880-3ca09c3ed778","Type":"ContainerStarted","Data":"e1092218f765999a454ffe2bb3df56b52c5df704603e2371a3f78b1b78f9c617"}
Sep 17 14:06:55.617070 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.617040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-46lvw" event={"ID":"8c3a0f5c-65b9-46f6-a43d-08b7b7427fef","Type":"ContainerStarted","Data":"2c46a53cc41d57061172a066eb1702c38bfeddda15b60cec1482704bef2ba3bf"}
Sep 17 14:06:55.619516 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.619490    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gt5cj" event={"ID":"260ac517-f5f9-421e-b392-3155fa34f1ae","Type":"ContainerStarted","Data":"4f2650ee5dc7476853cfa7dbf3a49e35299cc2aac1eb1aa62c803c9478dc16f3"}
Sep 17 14:06:55.624221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.624196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxm97" event={"ID":"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d","Type":"ContainerStarted","Data":"c5e411e813e4cd59ce4a223ecc6462ead21bdc76406b0c8e3e6c65f0063dc946"}
Sep 17 14:06:55.636148 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:55.636122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"c38b32708da242002cd9155b37b495baf82b57fb9a0a965bd2c5f7ae538d9922"}
Sep 17 14:06:56.023870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:56.023833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:56.024062 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:56.023932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:56.024135 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.024061    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:56.024135 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.024126    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret podName:c3a2822e-00c7-4ac7-85cc-816306696a1f nodeName:}" failed. No retries permitted until 2026-09-17 14:06:58.024106112 +0000 UTC m=+4.957611300 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret") pod "global-pull-secret-syncer-52tc9" (UID: "c3a2822e-00c7-4ac7-85cc-816306696a1f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:56.024575 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.024553    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:56.024710 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.024612    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs podName:e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d nodeName:}" failed. No retries permitted until 2026-09-17 14:06:58.02459677 +0000 UTC m=+4.958101943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs") pod "network-metrics-daemon-h5mnt" (UID: "e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:56.151218 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:56.151190    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 14:06:56.226731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:56.226044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:56.226731 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.226245    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:56.226731 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.226267    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:56.226731 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.226280    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wlrr4 for pod openshift-network-diagnostics/network-check-target-x8285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:56.226731 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.226356    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4 podName:5af832b9-d1bc-4c75-8703-dc0dfdd6b174 nodeName:}" failed. No retries permitted until 2026-09-17 14:06:58.226337103 +0000 UTC m=+5.159842291 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlrr4" (UniqueName: "kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4") pod "network-check-target-x8285" (UID: "5af832b9-d1bc-4c75-8703-dc0dfdd6b174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:56.503873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:56.503772    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 14:01:55 +0000 UTC" deadline="2028-03-16 20:00:07.354973255 +0000 UTC"
Sep 17 14:06:56.503873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:56.503817    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13109h53m10.851162088s"
Sep 17 14:06:56.513795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:56.513038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:56.513795 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.513189    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:06:56.513795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:56.513628    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:56.513795 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:56.513746    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:06:57.520289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:57.519762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:57.520289 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:57.520007    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:06:58.041827 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:58.041786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:58.042002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:58.041858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:58.042122 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.042011    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:58.042122 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.042074    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs podName:e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:02.042054969 +0000 UTC m=+8.975560147 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs") pod "network-metrics-daemon-h5mnt" (UID: "e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:06:58.042257 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.042142    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:58.042257 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.042190    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret podName:c3a2822e-00c7-4ac7-85cc-816306696a1f nodeName:}" failed. No retries permitted until 2026-09-17 14:07:02.04217578 +0000 UTC m=+8.975680969 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret") pod "global-pull-secret-syncer-52tc9" (UID: "c3a2822e-00c7-4ac7-85cc-816306696a1f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:06:58.243805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:58.243720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:58.243980 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.243917    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:06:58.243980 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.243944    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:06:58.243980 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.243960    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wlrr4 for pod openshift-network-diagnostics/network-check-target-x8285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:58.244144 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.244027    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4 podName:5af832b9-d1bc-4c75-8703-dc0dfdd6b174 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:02.244007675 +0000 UTC m=+9.177512862 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlrr4" (UniqueName: "kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4") pod "network-check-target-x8285" (UID: "5af832b9-d1bc-4c75-8703-dc0dfdd6b174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:06:58.513452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:58.513372    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:06:58.513624 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.513499    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:06:58.513624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:58.513376    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:06:58.513741 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:58.513662    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:06:59.514024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:06:59.513982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:06:59.514510 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:06:59.514127    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:00.513758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:00.513724    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:00.513960 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:00.513854    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:00.513960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:00.513918    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:00.514090 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:00.514045    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:01.513995 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:01.513718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:01.513995 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:01.513839    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:02.077439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:02.077158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:02.077439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:02.077215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:02.077439 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.077425    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:07:02.077647 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.077475    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:07:02.077647 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.077492    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs podName:e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:10.077472199 +0000 UTC m=+17.010977386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs") pod "network-metrics-daemon-h5mnt" (UID: "e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:07:02.077647 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.077514    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret podName:c3a2822e-00c7-4ac7-85cc-816306696a1f nodeName:}" failed. No retries permitted until 2026-09-17 14:07:10.077503577 +0000 UTC m=+17.011008754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret") pod "global-pull-secret-syncer-52tc9" (UID: "c3a2822e-00c7-4ac7-85cc-816306696a1f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:07:02.279052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:02.278965    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:02.279229 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.279155    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:07:02.279229 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.279179    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:07:02.279229 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.279194    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wlrr4 for pod openshift-network-diagnostics/network-check-target-x8285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:07:02.279421 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.279256    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4 podName:5af832b9-d1bc-4c75-8703-dc0dfdd6b174 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:10.279236282 +0000 UTC m=+17.212741462 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlrr4" (UniqueName: "kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4") pod "network-check-target-x8285" (UID: "5af832b9-d1bc-4c75-8703-dc0dfdd6b174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:07:02.513575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:02.513494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:02.513725 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:02.513494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:02.513725 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.513625    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:02.513725 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:02.513703    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:03.514876 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:03.514370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:03.514876 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:03.514483    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:04.513047 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:04.513019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:04.513193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:04.513019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:04.513193 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:04.513131    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:04.513334 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:04.513211    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:05.514105 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:05.514077    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:05.514583 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:05.514197    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:06.513960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:06.513931    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:06.514073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:06.513964    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:06.514073 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:06.514050    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:06.514454 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:06.514130    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:07.513994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:07.513953    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:07.514183 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:07.514093    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:08.513317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:08.513287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:08.513481 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:08.513295    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:08.513481 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:08.513423    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:08.513612 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:08.513486    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:09.513338 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:09.513309    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:09.513760 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:09.513411    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:10.140475 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:10.140438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:10.140676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:10.140499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:10.140676 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.140584    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:07:10.140676 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.140612    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:07:10.140676 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.140655    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret podName:c3a2822e-00c7-4ac7-85cc-816306696a1f nodeName:}" failed. No retries permitted until 2026-09-17 14:07:26.140630913 +0000 UTC m=+33.074136101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret") pod "global-pull-secret-syncer-52tc9" (UID: "c3a2822e-00c7-4ac7-85cc-816306696a1f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:07:10.140676 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.140675    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs podName:e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:26.140665747 +0000 UTC m=+33.074170919 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs") pod "network-metrics-daemon-h5mnt" (UID: "e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:07:10.341630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:10.341587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:10.341790 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.341736    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:07:10.341790 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.341759    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:07:10.341790 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.341770    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wlrr4 for pod openshift-network-diagnostics/network-check-target-x8285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:07:10.341958 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.341826    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4 podName:5af832b9-d1bc-4c75-8703-dc0dfdd6b174 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:26.341809229 +0000 UTC m=+33.275314413 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlrr4" (UniqueName: "kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4") pod "network-check-target-x8285" (UID: "5af832b9-d1bc-4c75-8703-dc0dfdd6b174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:07:10.513152 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:10.513080    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:10.513301 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.513204    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:10.513301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:10.513228    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:10.513753 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:10.513364    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:11.516307 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:11.516282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:11.516670 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:11.516388    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:12.513414 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:12.513377    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:12.513414 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:12.513414    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:12.513635 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:12.513520    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:12.513692 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:12.513655    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:13.513891 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:13.513533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:13.513891 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:13.513664    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:14.514121 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.513438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:14.514121 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:14.513896    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:14.514121 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.513976    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:14.514121 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:14.514074    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:14.677887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.677858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gt5cj" event={"ID":"260ac517-f5f9-421e-b392-3155fa34f1ae","Type":"ContainerStarted","Data":"54b5d744e5b58bf31ad7cfdc50a9efc47fc00a48bd6750246f372079b3f71ebd"}
Sep 17 14:07:14.679300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.679279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dxm97" event={"ID":"4d20f337-ad17-4d2c-a2d3-e4e7c15b162d","Type":"ContainerStarted","Data":"e30574ce6285dd0db4d7afc8f45513673f13b8528f7d963e9525ac59035c5fc1"}
Sep 17 14:07:14.684657 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.684633    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:07:14.685003 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.684983    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6762df1-b224-4466-ad96-74ee1af45c5f" containerID="b509c88d35c4524a9f95c345702eacf46a3a9acc358adc75a1249f6fba092ab2" exitCode=1
Sep 17 14:07:14.685073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.685046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"336b21c13c91a1f4e00f5e2532d3292373735b4e2ae30d14bdba5253b3184811"}
Sep 17 14:07:14.685134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.685121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"a2c4abc590e71dae04a6c89850299a7b6d31f442d5560b7e6b5c8211d30e7119"}
Sep 17 14:07:14.685179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.685166    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"92e676e98177113e0d6fd345123c9b7c56f5561722de8f71c1124d365d03a349"}
Sep 17 14:07:14.685210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.685196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"b843c7939bbe6294373e70085203f8bb295de06e21499353556db14bad4bb22c"}
Sep 17 14:07:14.685247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.685222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerDied","Data":"b509c88d35c4524a9f95c345702eacf46a3a9acc358adc75a1249f6fba092ab2"}
Sep 17 14:07:14.685310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.685279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"d7b0b0d655e66d8b39c79247f51576b6d83475629a2875be536b4c554dc34062"}
Sep 17 14:07:14.686465 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.686447    2565 generic.go:356] "Generic (PLEG): container finished" podID="886834cb-4fbd-4b7d-a883-0aa36121b1d6" containerID="aaa30ffcb8f30e5824fc3c02576a3aedb168e862d83acd2c82b61bf872f6d242" exitCode=0
Sep 17 14:07:14.686522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.686504    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" event={"ID":"886834cb-4fbd-4b7d-a883-0aa36121b1d6","Type":"ContainerDied","Data":"aaa30ffcb8f30e5824fc3c02576a3aedb168e862d83acd2c82b61bf872f6d242"}
Sep 17 14:07:14.689714 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.689693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5" event={"ID":"66d0451d-5e95-4e84-bf53-0df798ea36df","Type":"ContainerStarted","Data":"6d7f9b32635e6c24169e48a0fe3a17afc25ebab61597de35fc77ce073e8579ce"}
Sep 17 14:07:14.691702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.691683    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal" event={"ID":"4fd9853fb4ec76c374d0d4c456a7e8b2","Type":"ContainerStarted","Data":"06017a40a7cb07a0a56055b50153925221d6b5e25993042a2bf609a68b160640"}
Sep 17 14:07:14.693009 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.692985    2565 generic.go:356] "Generic (PLEG): container finished" podID="82c31071652d7cafc3e9e053617363c6" containerID="e38ef373eb5bacbd9163da2f26181e46780eb9b411e4343c321f3137a2235f31" exitCode=0
Sep 17 14:07:14.693082 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.693039    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal" event={"ID":"82c31071652d7cafc3e9e053617363c6","Type":"ContainerDied","Data":"e38ef373eb5bacbd9163da2f26181e46780eb9b411e4343c321f3137a2235f31"}
Sep 17 14:07:14.694643 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.694625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hztvs" event={"ID":"f2312773-bed0-40d9-a40f-1b8f822d4c98","Type":"ContainerStarted","Data":"5732e1b7a749e03227e90d7c84dde899532a062ecd896081ec5a52b0512b18ca"}
Sep 17 14:07:14.696091 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.696069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ggczn" event={"ID":"c41caec2-34bf-49de-a880-3ca09c3ed778","Type":"ContainerStarted","Data":"bbb48210497bf47809146a3d2729495ff7935bc7c9c33a79500f1d90fc4a9b92"}
Sep 17 14:07:14.697301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.697284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-46lvw" event={"ID":"8c3a0f5c-65b9-46f6-a43d-08b7b7427fef","Type":"ContainerStarted","Data":"df4c8bc080e22af3bfd9be285bcbfc782a3ee56ce73d1c69b9a1dbfbc86fde42"}
Sep 17 14:07:14.701447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.701413    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gt5cj" podStartSLOduration=1.35291141 podStartE2EDuration="20.701401619s" totalImagesPullingTime="19.348490209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:14.700162255 +0000 UTC m=+21.633667448" watchObservedRunningTime="2026-09-17 14:07:14.701401619 +0000 UTC m=+21.634906814"
Sep 17 14:07:14.743420 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.743380    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-175.ec2.internal" podStartSLOduration=19.743364751 podStartE2EDuration="19.743364751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:14.742987747 +0000 UTC m=+21.676492940" watchObservedRunningTime="2026-09-17 14:07:14.743364751 +0000 UTC m=+21.676869943"
Sep 17 14:07:14.809039 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.809000    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-46lvw" podStartSLOduration=2.085424198 podStartE2EDuration="20.808987093s" totalImagesPullingTime="18.723562895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:14.808877675 +0000 UTC m=+21.742382878" watchObservedRunningTime="2026-09-17 14:07:14.808987093 +0000 UTC m=+21.742492291"
Sep 17 14:07:14.809307 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.809286    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ggczn" podStartSLOduration=2.102761893 podStartE2EDuration="20.809279224s" totalImagesPullingTime="18.706517331s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:14.792239493 +0000 UTC m=+21.725744685" watchObservedRunningTime="2026-09-17 14:07:14.809279224 +0000 UTC m=+21.742784415"
Sep 17 14:07:14.824310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.824274    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dxm97" podStartSLOduration=2.178359846 podStartE2EDuration="20.824244024s" totalImagesPullingTime="18.645884178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:14.824204163 +0000 UTC m=+21.757709356" watchObservedRunningTime="2026-09-17 14:07:14.824244024 +0000 UTC m=+21.757749194"
Sep 17 14:07:14.861798 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.861776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:14.958797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:14.958777    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:15.064034 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.063979    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:15.459242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.459165    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:15.517377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.517307    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:15.518283 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:15.517416    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:15.558445 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.558413    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:15.701461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.701432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal" event={"ID":"82c31071652d7cafc3e9e053617363c6","Type":"ContainerStarted","Data":"13a53f1dcf86dbcf0b9be83fa7ce2236c1d763bc2e8710165d65e753d70d3690"}
Sep 17 14:07:15.703508 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.703481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qrls6" event={"ID":"21f87fcc-0335-4b5d-b1c0-65a17867de35","Type":"ContainerStarted","Data":"94cbb9ec89ff1d7bcf1666e8c8ac6dd47cd249c20eb27d511799882e45c8aeef"}
Sep 17 14:07:15.719302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.719203    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-175.ec2.internal" podStartSLOduration=20.719184077 podStartE2EDuration="20.719184077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:15.7188936 +0000 UTC m=+22.652398794" watchObservedRunningTime="2026-09-17 14:07:15.719184077 +0000 UTC m=+22.652689271"
Sep 17 14:07:15.719698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.719660    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hztvs" podStartSLOduration=3.053484725 podStartE2EDuration="21.719650033s" totalImagesPullingTime="18.666165308s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:14.841092336 +0000 UTC m=+21.774597526" watchObservedRunningTime="2026-09-17 14:07:15.719650033 +0000 UTC m=+22.653155227"
Sep 17 14:07:15.752532 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.752495    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qrls6" podStartSLOduration=3.102876922 podStartE2EDuration="21.752481934s" totalImagesPullingTime="18.649605012s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:15.752309002 +0000 UTC m=+22.685814195" watchObservedRunningTime="2026-09-17 14:07:15.752481934 +0000 UTC m=+22.685987125"
Sep 17 14:07:15.786505 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.786313    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 14:07:15.814038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.814017    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:07:15.814568 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.814549    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:07:15.881742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.881716    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:15.973514 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:15.973464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:16.463720 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.463627    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T14:07:15.786494328Z","UUID":"ddd1fcc9-77c0-4ffd-8932-9706ad22cd2e","Handler":null,"Name":"","Endpoint":""}
Sep 17 14:07:16.465327 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.465308    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 14:07:16.465426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.465340    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 14:07:16.513867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.513842    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:16.514022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.513879    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:16.514022 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:16.513963    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:16.514162 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:16.514093    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:16.710942 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.710877    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:07:16.711383 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.711339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"f0bbd38de8b9510cd0197019324bb05331f6072fe1e10db4426eb07cf9c3b9fc"}
Sep 17 14:07:16.713415 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.713389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5" event={"ID":"66d0451d-5e95-4e84-bf53-0df798ea36df","Type":"ContainerStarted","Data":"6e987fbe82fadfb717ea0cdd0d91edf23799e986345f250599470647bf1751cc"}
Sep 17 14:07:16.892502 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:16.892465    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:17.333916 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:17.333885    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:17.513466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:17.513436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:17.513625 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:17.513537    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:17.547547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:17.547504    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:17.717976 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:17.717900    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 14:07:17.717976 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:17.717905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5" event={"ID":"66d0451d-5e95-4e84-bf53-0df798ea36df","Type":"ContainerStarted","Data":"15862a6b792e6a38485cb3a3dffd413b3c6c631b3d27d99a97729f1c9f1b8c0f"}
Sep 17 14:07:17.752387 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:17.752339    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fwk5" podStartSLOduration=2.866835043 podStartE2EDuration="23.752325608s" totalImagesPullingTime="20.885490565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:17.752193228 +0000 UTC m=+24.685698419" watchObservedRunningTime="2026-09-17 14:07:17.752325608 +0000 UTC m=+24.685830800"
Sep 17 14:07:18.513072 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:18.513044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:18.513216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:18.513086    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:18.513216 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:18.513161    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:18.513341 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:18.513290    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:19.514269 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:19.514074    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:19.514749 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:19.514373    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:19.727058 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:19.727033    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:07:19.727577 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:19.727548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"0d8f003590f4eb258407d544bd1b9f0a647764c56a761fa207e4b3f43c11a8be"}
Sep 17 14:07:19.727850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:19.727816    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:07:19.728049 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:19.728027    2565 scope.go:117] "RemoveContainer" containerID="b509c88d35c4524a9f95c345702eacf46a3a9acc358adc75a1249f6fba092ab2"
Sep 17 14:07:19.730009 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:19.729982    2565 generic.go:356] "Generic (PLEG): container finished" podID="886834cb-4fbd-4b7d-a883-0aa36121b1d6" containerID="45ec29accb3affbfe4f2e7ef8ac891015ba1f60b758cff78c43a33acaf8a3e0f" exitCode=0
Sep 17 14:07:19.730119 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:19.730025    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" event={"ID":"886834cb-4fbd-4b7d-a883-0aa36121b1d6","Type":"ContainerDied","Data":"45ec29accb3affbfe4f2e7ef8ac891015ba1f60b758cff78c43a33acaf8a3e0f"}
Sep 17 14:07:19.746270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:19.746229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:07:20.179357 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.179289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:20.513816 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.513745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:20.513955 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:20.513869    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:20.513955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.513925    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:20.514066 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:20.514044    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:20.737488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.737463    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:07:20.738149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.737876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" event={"ID":"a6762df1-b224-4466-ad96-74ee1af45c5f","Type":"ContainerStarted","Data":"4b89a38be8d904b0257517ccec012cc5e8e552452f6728c09450f0bc881b5a19"}
Sep 17 14:07:20.738247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.738225    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:07:20.738337 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.738324    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:07:20.756174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.756151    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:07:20.803368 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.803203    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm" podStartSLOduration=8.076829667 podStartE2EDuration="26.803187425s" totalImagesPullingTime="18.726357758s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:20.785383855 +0000 UTC m=+27.718889052" watchObservedRunningTime="2026-09-17 14:07:20.803187425 +0000 UTC m=+27.736692617"
Sep 17 14:07:20.803578 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.803563    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x8285"]
Sep 17 14:07:20.803658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.803648    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:20.803733 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:20.803718    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:20.811358 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.811329    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h5mnt"]
Sep 17 14:07:20.811449 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.811397    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:20.811507 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:20.811474    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:20.813142 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.813104    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-52tc9"]
Sep 17 14:07:20.813242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:20.813184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:20.813321 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:20.813291    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:21.743179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:21.743140    2565 generic.go:356] "Generic (PLEG): container finished" podID="886834cb-4fbd-4b7d-a883-0aa36121b1d6" containerID="13a1c24cfa7d294de9ccb6d82c97a4190f39c084e9a35c636c82f574aad1ed9a" exitCode=0
Sep 17 14:07:21.743668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:21.743191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" event={"ID":"886834cb-4fbd-4b7d-a883-0aa36121b1d6","Type":"ContainerDied","Data":"13a1c24cfa7d294de9ccb6d82c97a4190f39c084e9a35c636c82f574aad1ed9a"}
Sep 17 14:07:21.868046 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:21.868017    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:21.982757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:21.982730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:22.513946 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:22.513873    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:22.514077 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:22.513960    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:22.514077 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:22.513979    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:22.514077 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:22.514000    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:22.514241 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:22.514114    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:22.514241 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:22.514204    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:23.985515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:23.985481    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:07:23.985898 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:23.985600    2565 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 14:07:23.986158 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:23.986138    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-46lvw"
Sep 17 14:07:24.055365 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:24.055337    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:24.513749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:24.513727    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:24.513749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:24.513745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:24.513897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:24.513730    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:24.513897 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:24.513843    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x8285" podUID="5af832b9-d1bc-4c75-8703-dc0dfdd6b174"
Sep 17 14:07:24.513968 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:24.513893    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-h5mnt" podUID="e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d"
Sep 17 14:07:24.513968 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:24.513959    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-52tc9" podUID="c3a2822e-00c7-4ac7-85cc-816306696a1f"
Sep 17 14:07:24.752595 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:24.752530    2565 generic.go:356] "Generic (PLEG): container finished" podID="886834cb-4fbd-4b7d-a883-0aa36121b1d6" containerID="6a084293eab3001acd210b1ab700c98f97b177f22178a55b91f30cd9b7b0e38e" exitCode=0
Sep 17 14:07:24.752706 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:24.752627    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" event={"ID":"886834cb-4fbd-4b7d-a883-0aa36121b1d6","Type":"ContainerDied","Data":"6a084293eab3001acd210b1ab700c98f97b177f22178a55b91f30cd9b7b0e38e"}
Sep 17 14:07:25.810625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.810599    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-175.ec2.internal" event="NodeReady"
Sep 17 14:07:25.810981 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.810716    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 14:07:25.899812 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.899743    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7wvpb"]
Sep 17 14:07:25.901562 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.901536    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7wvpb"
Sep 17 14:07:25.906918 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.906889    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 14:07:25.907018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.906938    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 14:07:25.907018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.906960    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 14:07:25.907018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.906960    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jltzw\""
Sep 17 14:07:25.912925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.912901    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-flvcb"]
Sep 17 14:07:25.914814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.914794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:25.919546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.919531    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 14:07:25.923052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.923032    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jdh9k\""
Sep 17 14:07:25.924118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:25.924103    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 14:07:26.023702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.023678    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7wvpb"]
Sep 17 14:07:26.024388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.024369    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-flvcb"]
Sep 17 14:07:26.068452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.068426    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwkfv\" (UniqueName: \"kubernetes.io/projected/3b860267-277a-4b1b-85b8-850e86d2cb3d-kube-api-access-mwkfv\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.068570 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.068460    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b860267-277a-4b1b-85b8-850e86d2cb3d-config-volume\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.068570 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.068530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s5pw\" (UniqueName: \"kubernetes.io/projected/3d84b0d1-637b-439f-a466-1d55f375e01f-kube-api-access-9s5pw\") pod \"ingress-canary-7wvpb\" (UID: \"3d84b0d1-637b-439f-a466-1d55f375e01f\") " pod="openshift-ingress-canary/ingress-canary-7wvpb"
Sep 17 14:07:26.068688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.068577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b860267-277a-4b1b-85b8-850e86d2cb3d-tmp-dir\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.068688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.068642    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d84b0d1-637b-439f-a466-1d55f375e01f-cert\") pod \"ingress-canary-7wvpb\" (UID: \"3d84b0d1-637b-439f-a466-1d55f375e01f\") " pod="openshift-ingress-canary/ingress-canary-7wvpb"
Sep 17 14:07:26.068688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.068658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.169179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:26.169179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d84b0d1-637b-439f-a466-1d55f375e01f-cert\") pod \"ingress-canary-7wvpb\" (UID: \"3d84b0d1-637b-439f-a466-1d55f375e01f\") " pod="openshift-ingress-canary/ingress-canary-7wvpb"
Sep 17 14:07:26.169453 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.169453 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.169293    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:07:26.169453 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.169370    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs podName:e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:58.169348661 +0000 UTC m=+65.102853845 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs") pod "network-metrics-daemon-h5mnt" (UID: "e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 14:07:26.169453 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.169373    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:26.169453 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwkfv\" (UniqueName: \"kubernetes.io/projected/3b860267-277a-4b1b-85b8-850e86d2cb3d-kube-api-access-mwkfv\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.169453 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.169427    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls podName:3b860267-277a-4b1b-85b8-850e86d2cb3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:26.669410339 +0000 UTC m=+33.602915521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls") pod "dns-default-flvcb" (UID: "3b860267-277a-4b1b-85b8-850e86d2cb3d") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:26.169749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b860267-277a-4b1b-85b8-850e86d2cb3d-config-volume\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.169749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s5pw\" (UniqueName: \"kubernetes.io/projected/3d84b0d1-637b-439f-a466-1d55f375e01f-kube-api-access-9s5pw\") pod \"ingress-canary-7wvpb\" (UID: \"3d84b0d1-637b-439f-a466-1d55f375e01f\") " pod="openshift-ingress-canary/ingress-canary-7wvpb"
Sep 17 14:07:26.169749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b860267-277a-4b1b-85b8-850e86d2cb3d-tmp-dir\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.169749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:26.169917 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.169805    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 14:07:26.169917 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.169858    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret podName:c3a2822e-00c7-4ac7-85cc-816306696a1f nodeName:}" failed. No retries permitted until 2026-09-17 14:07:58.169843295 +0000 UTC m=+65.103348479 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret") pod "global-pull-secret-syncer-52tc9" (UID: "c3a2822e-00c7-4ac7-85cc-816306696a1f") : object "kube-system"/"original-pull-secret" not registered
Sep 17 14:07:26.170020 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.169972    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b860267-277a-4b1b-85b8-850e86d2cb3d-tmp-dir\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.170148 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.170127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3b860267-277a-4b1b-85b8-850e86d2cb3d-config-volume\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.173821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.173706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d84b0d1-637b-439f-a466-1d55f375e01f-cert\") pod \"ingress-canary-7wvpb\" (UID: \"3d84b0d1-637b-439f-a466-1d55f375e01f\") " pod="openshift-ingress-canary/ingress-canary-7wvpb"
Sep 17 14:07:26.184574 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.184535    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwkfv\" (UniqueName: \"kubernetes.io/projected/3b860267-277a-4b1b-85b8-850e86d2cb3d-kube-api-access-mwkfv\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.188746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.188723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s5pw\" (UniqueName: \"kubernetes.io/projected/3d84b0d1-637b-439f-a466-1d55f375e01f-kube-api-access-9s5pw\") pod \"ingress-canary-7wvpb\" (UID: \"3d84b0d1-637b-439f-a466-1d55f375e01f\") " pod="openshift-ingress-canary/ingress-canary-7wvpb"
Sep 17 14:07:26.212088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.212072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7wvpb"
Sep 17 14:07:26.261403 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.261371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:26.371346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.371317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:26.371538 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.371515    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 14:07:26.371628 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.371545    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 14:07:26.371628 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.371560    2565 projected.go:196] Error preparing data for projected volume kube-api-access-wlrr4 for pod openshift-network-diagnostics/network-check-target-x8285: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:07:26.371628 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.371621    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4 podName:5af832b9-d1bc-4c75-8703-dc0dfdd6b174 nodeName:}" failed. No retries permitted until 2026-09-17 14:07:58.371604353 +0000 UTC m=+65.305109546 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wlrr4" (UniqueName: "kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4") pod "network-check-target-x8285" (UID: "5af832b9-d1bc-4c75-8703-dc0dfdd6b174") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 14:07:26.514103 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.514042    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:26.514232 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.514042    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:26.514314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.514042    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:26.524090 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.524044    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 14:07:26.524191 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.524109    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c2694\""
Sep 17 14:07:26.524191 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.524175    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ck6nq\""
Sep 17 14:07:26.524815 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.524797    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 14:07:26.524937 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.524805    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 14:07:26.525597 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.525576    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7wvpb"]
Sep 17 14:07:26.528028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.527971    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 14:07:26.530343 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:07:26.530322    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d84b0d1_637b_439f_a466_1d55f375e01f.slice/crio-65514ef35ef1210b7b28e56bdb86cdd9bd0b22440fa114082f49739579f5fdab WatchSource:0}: Error finding container 65514ef35ef1210b7b28e56bdb86cdd9bd0b22440fa114082f49739579f5fdab: Status 404 returned error can't find the container with id 65514ef35ef1210b7b28e56bdb86cdd9bd0b22440fa114082f49739579f5fdab
Sep 17 14:07:26.672958 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.672932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:26.673113 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.673089    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:26.673230 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:26.673162    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls podName:3b860267-277a-4b1b-85b8-850e86d2cb3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:27.673142995 +0000 UTC m=+34.606648188 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls") pod "dns-default-flvcb" (UID: "3b860267-277a-4b1b-85b8-850e86d2cb3d") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:26.758748 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:26.758717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7wvpb" event={"ID":"3d84b0d1-637b-439f-a466-1d55f375e01f","Type":"ContainerStarted","Data":"65514ef35ef1210b7b28e56bdb86cdd9bd0b22440fa114082f49739579f5fdab"}
Sep 17 14:07:27.626338 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:27.626303    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:27.681207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:27.681180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:27.681384 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:27.681312    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:27.681384 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:27.681377    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls podName:3b860267-277a-4b1b-85b8-850e86d2cb3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:29.681358322 +0000 UTC m=+36.614863517 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls") pod "dns-default-flvcb" (UID: "3b860267-277a-4b1b-85b8-850e86d2cb3d") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:27.725733 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:27.725693    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:29.694462 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:29.694426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:29.695010 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:29.694545    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:29.695010 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:29.694607    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls podName:3b860267-277a-4b1b-85b8-850e86d2cb3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:33.694590088 +0000 UTC m=+40.628095274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls") pod "dns-default-flvcb" (UID: "3b860267-277a-4b1b-85b8-850e86d2cb3d") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:30.763743 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:30.763663    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:30.770842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:30.770819    2565 generic.go:356] "Generic (PLEG): container finished" podID="886834cb-4fbd-4b7d-a883-0aa36121b1d6" containerID="cb6c172e6cdde0077b0486f920dc2a101996a270c0367ca1e819570c4c618dc3" exitCode=0
Sep 17 14:07:30.770943 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:30.770892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" event={"ID":"886834cb-4fbd-4b7d-a883-0aa36121b1d6","Type":"ContainerDied","Data":"cb6c172e6cdde0077b0486f920dc2a101996a270c0367ca1e819570c4c618dc3"}
Sep 17 14:07:30.772466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:30.772444    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7wvpb" event={"ID":"3d84b0d1-637b-439f-a466-1d55f375e01f","Type":"ContainerStarted","Data":"3ba82fa426ce594a0f49c02d182286267c132f01514f5ea04a1ebaae64763f78"}
Sep 17 14:07:30.828088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:30.828043    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7wvpb" podStartSLOduration=2.443710931 podStartE2EDuration="5.828023504s" totalImagesPullingTime="3.384312573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:30.827995872 +0000 UTC m=+37.761501065" watchObservedRunningTime="2026-09-17 14:07:30.828023504 +0000 UTC m=+37.761528697"
Sep 17 14:07:30.906948 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:30.906918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:31.513702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:31.513644    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:31.613346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:31.613321    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:31.777978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:31.777923    2565 generic.go:356] "Generic (PLEG): container finished" podID="886834cb-4fbd-4b7d-a883-0aa36121b1d6" containerID="c9f268414727d70fdb100598b78ce461f4a5fee9f4474e413b2900099767e255" exitCode=0
Sep 17 14:07:31.778378 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:31.778021    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" event={"ID":"886834cb-4fbd-4b7d-a883-0aa36121b1d6","Type":"ContainerDied","Data":"c9f268414727d70fdb100598b78ce461f4a5fee9f4474e413b2900099767e255"}
Sep 17 14:07:32.333625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:32.333594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:32.432821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:32.432776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:32.784066 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:32.783998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" event={"ID":"886834cb-4fbd-4b7d-a883-0aa36121b1d6","Type":"ContainerStarted","Data":"e8b05e6c37ac24a838131841046922a80b9e1a9613bcaa3361ac7266b42ca679"}
Sep 17 14:07:32.809752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:32.809703    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7vlwh" podStartSLOduration=7.158727172 podStartE2EDuration="38.80968985s" totalImagesPullingTime="31.650962678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:07:32.80926455 +0000 UTC m=+39.742769733" watchObservedRunningTime="2026-09-17 14:07:32.80968985 +0000 UTC m=+39.743195042"
Sep 17 14:07:32.948531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:32.948495    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:33.023632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:33.023604    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:33.096713 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:33.096677    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:33.721190 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:33.721156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:33.721324 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:33.721309    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:33.721377 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:33.721365    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls podName:3b860267-277a-4b1b-85b8-850e86d2cb3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:41.721347622 +0000 UTC m=+48.654852816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls") pod "dns-default-flvcb" (UID: "3b860267-277a-4b1b-85b8-850e86d2cb3d") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:35.672860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:35.672825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:35.768850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:35.768820    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:41.771780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:41.771746    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:41.772166 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:41.771843    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:41.772166 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:41.771893    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls podName:3b860267-277a-4b1b-85b8-850e86d2cb3d nodeName:}" failed. No retries permitted until 2026-09-17 14:07:57.771879527 +0000 UTC m=+64.705384696 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls") pod "dns-default-flvcb" (UID: "3b860267-277a-4b1b-85b8-850e86d2cb3d") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:52.510387 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:52.510348    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:52.573297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:52.573268    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:52.756387 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:52.756361    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8f6tm"
Sep 17 14:07:53.035471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:53.035439    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:53.209993 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:53.209965    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:07:57.871788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:57.871758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:07:57.872202 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:57.871863    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:07:57.872202 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:57.871907    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls podName:3b860267-277a-4b1b-85b8-850e86d2cb3d nodeName:}" failed. No retries permitted until 2026-09-17 14:08:29.871894916 +0000 UTC m=+96.805400110 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls") pod "dns-default-flvcb" (UID: "3b860267-277a-4b1b-85b8-850e86d2cb3d") : secret "dns-default-metrics-tls" not found
Sep 17 14:07:58.174873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.174811    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:58.174873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.174850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:07:58.178585 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.178564    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 14:07:58.178644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.178564    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 14:07:58.185628 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:58.185607    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 14:07:58.185709 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:07:58.185659    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs podName:e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d nodeName:}" failed. No retries permitted until 2026-09-17 14:09:02.185644377 +0000 UTC m=+129.119149551 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs") pod "network-metrics-daemon-h5mnt" (UID: "e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d") : secret "metrics-daemon-secret" not found
Sep 17 14:07:58.188744 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.188724    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c3a2822e-00c7-4ac7-85cc-816306696a1f-original-pull-secret\") pod \"global-pull-secret-syncer-52tc9\" (UID: \"c3a2822e-00c7-4ac7-85cc-816306696a1f\") " pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:58.327868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.327846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-52tc9"
Sep 17 14:07:58.376624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.376590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:58.379248 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.379228    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 14:07:58.389884 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.389865    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 14:07:58.399615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.399590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlrr4\" (UniqueName: \"kubernetes.io/projected/5af832b9-d1bc-4c75-8703-dc0dfdd6b174-kube-api-access-wlrr4\") pod \"network-check-target-x8285\" (UID: \"5af832b9-d1bc-4c75-8703-dc0dfdd6b174\") " pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:58.632956 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.632536    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-52tc9"]
Sep 17 14:07:58.637181 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:07:58.637152    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a2822e_00c7_4ac7_85cc_816306696a1f.slice/crio-e0c6e1c4da03dd88f7e43b3b1bfc90bafc777cd7e9f86501726c6dc7b1d2cfbb WatchSource:0}: Error finding container e0c6e1c4da03dd88f7e43b3b1bfc90bafc777cd7e9f86501726c6dc7b1d2cfbb: Status 404 returned error can't find the container with id e0c6e1c4da03dd88f7e43b3b1bfc90bafc777cd7e9f86501726c6dc7b1d2cfbb
Sep 17 14:07:58.644039 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.644021    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c2694\""
Sep 17 14:07:58.652316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.652297    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:07:58.842289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.842248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-52tc9" event={"ID":"c3a2822e-00c7-4ac7-85cc-816306696a1f","Type":"ContainerStarted","Data":"e0c6e1c4da03dd88f7e43b3b1bfc90bafc777cd7e9f86501726c6dc7b1d2cfbb"}
Sep 17 14:07:58.966328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:58.966268    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x8285"]
Sep 17 14:07:58.970279 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:07:58.970242    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af832b9_d1bc_4c75_8703_dc0dfdd6b174.slice/crio-499abf00ec8c4cbcc75b562b3d4c56c876768a375fd442ff29d701a9a292d76c WatchSource:0}: Error finding container 499abf00ec8c4cbcc75b562b3d4c56c876768a375fd442ff29d701a9a292d76c: Status 404 returned error can't find the container with id 499abf00ec8c4cbcc75b562b3d4c56c876768a375fd442ff29d701a9a292d76c
Sep 17 14:07:59.846588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:07:59.846545    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x8285" event={"ID":"5af832b9-d1bc-4c75-8703-dc0dfdd6b174","Type":"ContainerStarted","Data":"499abf00ec8c4cbcc75b562b3d4c56c876768a375fd442ff29d701a9a292d76c"}
Sep 17 14:08:03.860179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:03.860138    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-52tc9" event={"ID":"c3a2822e-00c7-4ac7-85cc-816306696a1f","Type":"ContainerStarted","Data":"e2a1eb4d5d6b14fd5261b38efd98d3464863e3e8a1e84c3b5dfad9291d981ac2"}
Sep 17 14:08:03.861838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:03.861804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x8285" event={"ID":"5af832b9-d1bc-4c75-8703-dc0dfdd6b174","Type":"ContainerStarted","Data":"f5c9e97dcacdcdbfed8817d78f647cd626e8d21d04f945559874373fa4a44b25"}
Sep 17 14:08:03.861949 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:03.861893    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:08:03.881583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:03.881535    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-52tc9" podStartSLOduration=65.043600729 podStartE2EDuration="1m9.881518944s" totalImagesPullingTime="4.837918215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:08:03.88036237 +0000 UTC m=+70.813867562" watchObservedRunningTime="2026-09-17 14:08:03.881518944 +0000 UTC m=+70.815024136"
Sep 17 14:08:03.900699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:03.900647    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x8285" podStartSLOduration=65.398941731 podStartE2EDuration="1m9.900632947s" totalImagesPullingTime="4.501691216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:08:03.899632848 +0000 UTC m=+70.833138039" watchObservedRunningTime="2026-09-17 14:08:03.900632947 +0000 UTC m=+70.834138139"
Sep 17 14:08:29.955909 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:29.955869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:08:29.956468 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:29.955990    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 14:08:29.956468 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:29.956054    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls podName:3b860267-277a-4b1b-85b8-850e86d2cb3d nodeName:}" failed. No retries permitted until 2026-09-17 14:09:33.95604064 +0000 UTC m=+160.889545810 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls") pod "dns-default-flvcb" (UID: "3b860267-277a-4b1b-85b8-850e86d2cb3d") : secret "dns-default-metrics-tls" not found
Sep 17 14:08:34.867071 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:34.867034    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x8285"
Sep 17 14:08:44.106305 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.106069    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687"]
Sep 17 14:08:44.110737 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.110707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687"
Sep 17 14:08:44.113388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.113363    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 17 14:08:44.114055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.114037    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:08:44.114107 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.114059    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-vzflm\""
Sep 17 14:08:44.118935 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.118906    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687"]
Sep 17 14:08:44.141135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.141107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfsqj\" (UniqueName: \"kubernetes.io/projected/7ed7979b-5dea-4ff5-bc94-1ebb231696d7-kube-api-access-wfsqj\") pod \"volume-data-source-validator-86695b7cc7-49687\" (UID: \"7ed7979b-5dea-4ff5-bc94-1ebb231696d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687"
Sep 17 14:08:44.211239 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.211206    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"]
Sep 17 14:08:44.213052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.213032    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.218262 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.218231    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 17 14:08:44.218372 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.218281    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 17 14:08:44.218563 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.218544    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 17 14:08:44.218750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.218639    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-shv9p\""
Sep 17 14:08:44.218750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.218653    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:08:44.225099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.225077    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-fbr4k"]
Sep 17 14:08:44.226690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.226675    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.233208 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.233189    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 17 14:08:44.233335 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.233316    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 14:08:44.233535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.233516    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9zgrm\""
Sep 17 14:08:44.234698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.234683    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 14:08:44.234936 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.234922    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 17 14:08:44.239495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.239477    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 17 14:08:44.241356 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.241332    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfsqj\" (UniqueName: \"kubernetes.io/projected/7ed7979b-5dea-4ff5-bc94-1ebb231696d7-kube-api-access-wfsqj\") pod \"volume-data-source-validator-86695b7cc7-49687\" (UID: \"7ed7979b-5dea-4ff5-bc94-1ebb231696d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687"
Sep 17 14:08:44.241441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.241369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2zv\" (UniqueName: \"kubernetes.io/projected/ebf0ae63-0557-403a-a3ee-13969694249d-kube-api-access-7p2zv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.241441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.241388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf0ae63-0557-403a-a3ee-13969694249d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.241441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.241411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebf0ae63-0557-403a-a3ee-13969694249d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.243106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.243089    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"]
Sep 17 14:08:44.243743 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.243726    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-fbr4k"]
Sep 17 14:08:44.264679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.264658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfsqj\" (UniqueName: \"kubernetes.io/projected/7ed7979b-5dea-4ff5-bc94-1ebb231696d7-kube-api-access-wfsqj\") pod \"volume-data-source-validator-86695b7cc7-49687\" (UID: \"7ed7979b-5dea-4ff5-bc94-1ebb231696d7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687"
Sep 17 14:08:44.313551 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.313529    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dvbhf"]
Sep 17 14:08:44.315287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.315272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.316664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.316648    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7d847c5fc8-txhzg"]
Sep 17 14:08:44.318291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.318277    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.319496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.319480    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 17 14:08:44.319906 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.319891    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 17 14:08:44.320366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.320349    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 17 14:08:44.320457 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.320438    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:08:44.323856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.323833    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 17 14:08:44.323949 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.323924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 17 14:08:44.324048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.324029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-xfplb\""
Sep 17 14:08:44.324112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.324060    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 14:08:44.324168 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.324124    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 14:08:44.324402 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.324383    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 17 14:08:44.324521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.324409    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 17 14:08:44.328273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.328242    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hc8ht\""
Sep 17 14:08:44.332861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.332843    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 17 14:08:44.338359 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.338341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dvbhf"]
Sep 17 14:08:44.341812 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.341793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c4e844-c224-42f8-9d28-6f5c733038e8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.341895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.341819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk299\" (UniqueName: \"kubernetes.io/projected/50c4e844-c224-42f8-9d28-6f5c733038e8-kube-api-access-zk299\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.341895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.341836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-default-certificate\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.341895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.341853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjsxv\" (UniqueName: \"kubernetes.io/projected/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-kube-api-access-vjsxv\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.341895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.341870    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-stats-auth\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.342081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.341908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.342081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.341950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.342081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.341988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2zv\" (UniqueName: \"kubernetes.io/projected/ebf0ae63-0557-403a-a3ee-13969694249d-kube-api-access-7p2zv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.342081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342013    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-serving-cert\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.342081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342063    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf0ae63-0557-403a-a3ee-13969694249d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.342313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebf0ae63-0557-403a-a3ee-13969694249d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.342313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq69q\" (UniqueName: \"kubernetes.io/projected/183850d9-742f-4556-966d-99d1df6cc9a1-kube-api-access-qq69q\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.342313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/50c4e844-c224-42f8-9d28-6f5c733038e8-snapshots\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.342313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c4e844-c224-42f8-9d28-6f5c733038e8-service-ca-bundle\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.342313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342177    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-config\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.342313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50c4e844-c224-42f8-9d28-6f5c733038e8-tmp\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.342313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50c4e844-c224-42f8-9d28-6f5c733038e8-serving-cert\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.342313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-trusted-ca\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.342737 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.342543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ebf0ae63-0557-403a-a3ee-13969694249d-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.344273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.344235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ebf0ae63-0557-403a-a3ee-13969694249d-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.344695 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.344670    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d847c5fc8-txhzg"]
Sep 17 14:08:44.351878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.351861    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2zv\" (UniqueName: \"kubernetes.io/projected/ebf0ae63-0557-403a-a3ee-13969694249d-kube-api-access-7p2zv\") pod \"kube-storage-version-migrator-operator-6c8f84f897-chdpt\" (UID: \"ebf0ae63-0557-403a-a3ee-13969694249d\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.420949 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.420893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687"
Sep 17 14:08:44.442878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.442856    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/50c4e844-c224-42f8-9d28-6f5c733038e8-snapshots\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.442978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.442891    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c4e844-c224-42f8-9d28-6f5c733038e8-service-ca-bundle\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.442978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.442913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-config\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.442978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.442953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50c4e844-c224-42f8-9d28-6f5c733038e8-tmp\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.442978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.442969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50c4e844-c224-42f8-9d28-6f5c733038e8-serving-cert\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.443166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.442984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-trusted-ca\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.443166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443009    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c4e844-c224-42f8-9d28-6f5c733038e8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.443166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk299\" (UniqueName: \"kubernetes.io/projected/50c4e844-c224-42f8-9d28-6f5c733038e8-kube-api-access-zk299\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.443166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443056    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-default-certificate\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.443166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjsxv\" (UniqueName: \"kubernetes.io/projected/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-kube-api-access-vjsxv\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.443166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443106    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-stats-auth\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.443166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.443505 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443173    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.443505 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-serving-cert\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.443505 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq69q\" (UniqueName: \"kubernetes.io/projected/183850d9-742f-4556-966d-99d1df6cc9a1-kube-api-access-qq69q\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.443505 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443480    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50c4e844-c224-42f8-9d28-6f5c733038e8-tmp\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.443719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443530    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/50c4e844-c224-42f8-9d28-6f5c733038e8-snapshots\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.443719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.443610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c4e844-c224-42f8-9d28-6f5c733038e8-service-ca-bundle\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.444392 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:44.443962    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 14:08:44.444392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.444014    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-config\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.444392 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:44.444047    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:44.94402933 +0000 UTC m=+111.877534517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : secret "router-metrics-certs-default" not found
Sep 17 14:08:44.444392 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:44.444152    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:44.9441357 +0000 UTC m=+111.877640888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : configmap references non-existent config key: service-ca.crt
Sep 17 14:08:44.444392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.444044    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-trusted-ca\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.444765 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.444399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50c4e844-c224-42f8-9d28-6f5c733038e8-trusted-ca-bundle\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.445995 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.445976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/50c4e844-c224-42f8-9d28-6f5c733038e8-serving-cert\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.446324 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.446305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-serving-cert\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.446402 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.446386    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-stats-auth\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.446729 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.446711    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-default-certificate\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.452708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.452689    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq69q\" (UniqueName: \"kubernetes.io/projected/183850d9-742f-4556-966d-99d1df6cc9a1-kube-api-access-qq69q\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.453093 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.453072    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjsxv\" (UniqueName: \"kubernetes.io/projected/73c3fc4e-356f-49f8-a20d-18abf2d5c52a-kube-api-access-vjsxv\") pod \"console-operator-696884fb9c-dvbhf\" (UID: \"73c3fc4e-356f-49f8-a20d-18abf2d5c52a\") " pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.453403 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.453386    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk299\" (UniqueName: \"kubernetes.io/projected/50c4e844-c224-42f8-9d28-6f5c733038e8-kube-api-access-zk299\") pod \"insights-operator-5844b856db-fbr4k\" (UID: \"50c4e844-c224-42f8-9d28-6f5c733038e8\") " pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.521200 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.521161    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"
Sep 17 14:08:44.534184 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.534164    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-fbr4k"
Sep 17 14:08:44.554198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.554136    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:44.624610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.624582    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:44.702413 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.702343    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687"]
Sep 17 14:08:44.705795 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:08:44.705771    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed7979b_5dea_4ff5_bc94_1ebb231696d7.slice/crio-025c8567516d025c807808f1740fa41af1e0bd42b9bb553ab673cbb93abe78f9 WatchSource:0}: Error finding container 025c8567516d025c807808f1740fa41af1e0bd42b9bb553ab673cbb93abe78f9: Status 404 returned error can't find the container with id 025c8567516d025c807808f1740fa41af1e0bd42b9bb553ab673cbb93abe78f9
Sep 17 14:08:44.800438 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.800415    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt"]
Sep 17 14:08:44.803129 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:08:44.803104    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebf0ae63_0557_403a_a3ee_13969694249d.slice/crio-c67b5ef5d1fe6904fc918c4e9560a5a609c2f7413fc6b9e9117f747aeb49d860 WatchSource:0}: Error finding container c67b5ef5d1fe6904fc918c4e9560a5a609c2f7413fc6b9e9117f747aeb49d860: Status 404 returned error can't find the container with id c67b5ef5d1fe6904fc918c4e9560a5a609c2f7413fc6b9e9117f747aeb49d860
Sep 17 14:08:44.820023 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.820003    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-fbr4k"]
Sep 17 14:08:44.822643 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:08:44.822619    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c4e844_c224_42f8_9d28_6f5c733038e8.slice/crio-54a6c9d3d716290290749f89fc4719eccbc89eec61777480dcbecbab414b1cc5 WatchSource:0}: Error finding container 54a6c9d3d716290290749f89fc4719eccbc89eec61777480dcbecbab414b1cc5: Status 404 returned error can't find the container with id 54a6c9d3d716290290749f89fc4719eccbc89eec61777480dcbecbab414b1cc5
Sep 17 14:08:44.901641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.901621    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-dvbhf"]
Sep 17 14:08:44.904501 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:08:44.904479    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c3fc4e_356f_49f8_a20d_18abf2d5c52a.slice/crio-cab44092dc1172e5b6ae14d4e54bd48ab60549139af79d4ce52ba0e36388f455 WatchSource:0}: Error finding container cab44092dc1172e5b6ae14d4e54bd48ab60549139af79d4ce52ba0e36388f455: Status 404 returned error can't find the container with id cab44092dc1172e5b6ae14d4e54bd48ab60549139af79d4ce52ba0e36388f455
Sep 17 14:08:44.947412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.947386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.947503 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.947422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:44.947563 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:44.947538    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:45.947520645 +0000 UTC m=+112.881025835 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : configmap references non-existent config key: service-ca.crt
Sep 17 14:08:44.947622 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:44.947576    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 14:08:44.947671 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:44.947634    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:45.947617272 +0000 UTC m=+112.881122461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : secret "router-metrics-certs-default" not found
Sep 17 14:08:44.960838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.960778    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf" event={"ID":"73c3fc4e-356f-49f8-a20d-18abf2d5c52a","Type":"ContainerStarted","Data":"cab44092dc1172e5b6ae14d4e54bd48ab60549139af79d4ce52ba0e36388f455"}
Sep 17 14:08:44.961793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.961774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt" event={"ID":"ebf0ae63-0557-403a-a3ee-13969694249d","Type":"ContainerStarted","Data":"c67b5ef5d1fe6904fc918c4e9560a5a609c2f7413fc6b9e9117f747aeb49d860"}
Sep 17 14:08:44.962756 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.962738    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fbr4k" event={"ID":"50c4e844-c224-42f8-9d28-6f5c733038e8","Type":"ContainerStarted","Data":"54a6c9d3d716290290749f89fc4719eccbc89eec61777480dcbecbab414b1cc5"}
Sep 17 14:08:44.963862 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:44.963840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687" event={"ID":"7ed7979b-5dea-4ff5-bc94-1ebb231696d7","Type":"ContainerStarted","Data":"025c8567516d025c807808f1740fa41af1e0bd42b9bb553ab673cbb93abe78f9"}
Sep 17 14:08:45.954088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:45.954026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:45.954088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:45.954082    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:45.954501 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:45.954126    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:47.954109597 +0000 UTC m=+114.887614784 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : configmap references non-existent config key: service-ca.crt
Sep 17 14:08:45.954501 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:45.954195    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 14:08:45.954501 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:45.954244    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:47.954229604 +0000 UTC m=+114.887734791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : secret "router-metrics-certs-default" not found
Sep 17 14:08:45.969829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:45.969804    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687" event={"ID":"7ed7979b-5dea-4ff5-bc94-1ebb231696d7","Type":"ContainerStarted","Data":"7851d7baf4a3327240eb51e2faf9acdd0526c209d679b956a12866312132f1c5"}
Sep 17 14:08:45.988469 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:45.988416    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-49687" podStartSLOduration=0.851449551 podStartE2EDuration="1.988397296s" totalImagesPullingTime="1.136947745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:08:45.987121544 +0000 UTC m=+112.920626739" watchObservedRunningTime="2026-09-17 14:08:45.988397296 +0000 UTC m=+112.921902490"
Sep 17 14:08:47.969214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:47.969178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:47.969214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:47.969218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:47.969670 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:47.969360    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:51.969343208 +0000 UTC m=+118.902848383 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : configmap references non-existent config key: service-ca.crt
Sep 17 14:08:47.969670 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:47.969371    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 14:08:47.969670 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:47.969437    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:08:51.969420633 +0000 UTC m=+118.902925823 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : secret "router-metrics-certs-default" not found
Sep 17 14:08:48.980628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:48.980598    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/0.log"
Sep 17 14:08:48.981029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:48.980646    2565 generic.go:356] "Generic (PLEG): container finished" podID="73c3fc4e-356f-49f8-a20d-18abf2d5c52a" containerID="5dd37ec7ed356d67f17db2227071f39104b49a8e3970718c158da48e38c76d86" exitCode=255
Sep 17 14:08:48.981029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:48.980727    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf" event={"ID":"73c3fc4e-356f-49f8-a20d-18abf2d5c52a","Type":"ContainerDied","Data":"5dd37ec7ed356d67f17db2227071f39104b49a8e3970718c158da48e38c76d86"}
Sep 17 14:08:48.981029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:48.980901    2565 scope.go:117] "RemoveContainer" containerID="5dd37ec7ed356d67f17db2227071f39104b49a8e3970718c158da48e38c76d86"
Sep 17 14:08:48.982470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:48.982443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt" event={"ID":"ebf0ae63-0557-403a-a3ee-13969694249d","Type":"ContainerStarted","Data":"4e9fecd8cd53b32cbff6ac8f52884e7c6bc1d4eef6644731b0947ad75e99bcc3"}
Sep 17 14:08:48.984003 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:48.983982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fbr4k" event={"ID":"50c4e844-c224-42f8-9d28-6f5c733038e8","Type":"ContainerStarted","Data":"6fcff2e07074f447b39cb2b091bf46955393f36c54e322e6ae6e04250d2c7660"}
Sep 17 14:08:49.052944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.052890    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt" podStartSLOduration=1.852741211 podStartE2EDuration="5.052871772s" totalImagesPullingTime="3.200130561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:08:49.052227325 +0000 UTC m=+115.985732592" watchObservedRunningTime="2026-09-17 14:08:49.052871772 +0000 UTC m=+115.986376966"
Sep 17 14:08:49.119059 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.119014    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-fbr4k" podStartSLOduration=1.938436786 podStartE2EDuration="5.118997698s" totalImagesPullingTime="3.180560912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:08:49.117811934 +0000 UTC m=+116.051317126" watchObservedRunningTime="2026-09-17 14:08:49.118997698 +0000 UTC m=+116.052502944"
Sep 17 14:08:49.344094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.344059    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mtkwv"]
Sep 17 14:08:49.346416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.346395    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.348909 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.348884    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 14:08:49.349293 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.349246    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmmrx\""
Sep 17 14:08:49.349376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.349316    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 14:08:49.379457 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.379423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.379558 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.379494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48brp\" (UniqueName: \"kubernetes.io/projected/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-api-access-48brp\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.379616 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.379554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85f6b33e-7505-4478-907f-f75fa8d8b07a-crio-socket\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.379616 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.379581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.379695 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.379626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f6b33e-7505-4478-907f-f75fa8d8b07a-data-volume\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.450227 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.450205    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mtkwv"]
Sep 17 14:08:49.480171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.480147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48brp\" (UniqueName: \"kubernetes.io/projected/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-api-access-48brp\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.480281 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.480175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85f6b33e-7505-4478-907f-f75fa8d8b07a-crio-socket\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.480281 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.480193    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.480281 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.480215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f6b33e-7505-4478-907f-f75fa8d8b07a-data-volume\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.480443 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.480292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.480443 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.480309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85f6b33e-7505-4478-907f-f75fa8d8b07a-crio-socket\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.480443 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.480313    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:49.480443 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.480388    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:08:49.980365794 +0000 UTC m=+116.913870981 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:49.480443 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.480432    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:49.480608 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.480514    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:08:49.980498244 +0000 UTC m=+116.914003426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:49.480608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.480583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85f6b33e-7505-4478-907f-f75fa8d8b07a-data-volume\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.512665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.512639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48brp\" (UniqueName: \"kubernetes.io/projected/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-api-access-48brp\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.798307 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.798226    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:49.984246 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.984213    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.984645 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.984321    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:49.984645 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.984329    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:49.984645 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.984395    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:08:50.984381197 +0000 UTC m=+117.917886394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:49.984645 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.984449    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:49.984645 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.984508    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:08:50.984491646 +0000 UTC m=+117.917996834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:49.987989 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.987971    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:08:49.988418 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.988396    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/0.log"
Sep 17 14:08:49.988489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.988431    2565 generic.go:356] "Generic (PLEG): container finished" podID="73c3fc4e-356f-49f8-a20d-18abf2d5c52a" containerID="c768691d2f590a099b2f1d34707e165e78da87977c1df2181d78701e594cdd0b" exitCode=255
Sep 17 14:08:49.988542 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.988506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf" event={"ID":"73c3fc4e-356f-49f8-a20d-18abf2d5c52a","Type":"ContainerDied","Data":"c768691d2f590a099b2f1d34707e165e78da87977c1df2181d78701e594cdd0b"}
Sep 17 14:08:49.988584 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.988547    2565 scope.go:117] "RemoveContainer" containerID="5dd37ec7ed356d67f17db2227071f39104b49a8e3970718c158da48e38c76d86"
Sep 17 14:08:49.988760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:49.988736    2565 scope.go:117] "RemoveContainer" containerID="c768691d2f590a099b2f1d34707e165e78da87977c1df2181d78701e594cdd0b"
Sep 17 14:08:49.988913 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:49.988895    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dvbhf_openshift-console-operator(73c3fc4e-356f-49f8-a20d-18abf2d5c52a)\"" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf" podUID="73c3fc4e-356f-49f8-a20d-18abf2d5c52a"
Sep 17 14:08:50.032394 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:50.032374    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:50.216939 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:50.216900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:50.875538 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:50.875505    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:50.976581 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:50.976551    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:50.992883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:50.992859    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:08:50.993314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:50.993282    2565 scope.go:117] "RemoveContainer" containerID="c768691d2f590a099b2f1d34707e165e78da87977c1df2181d78701e594cdd0b"
Sep 17 14:08:50.993501 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:50.993475    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dvbhf_openshift-console-operator(73c3fc4e-356f-49f8-a20d-18abf2d5c52a)\"" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf" podUID="73c3fc4e-356f-49f8-a20d-18abf2d5c52a"
Sep 17 14:08:50.993899 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:50.993876    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:50.993994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:50.993937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:50.994068 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:50.994005    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:50.994068 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:50.994026    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:50.994158 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:50.994079    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:08:52.994058162 +0000 UTC m=+119.927563386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:50.994158 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:50.994101    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:08:52.994090434 +0000 UTC m=+119.927595606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:51.432445 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:51.432423    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxm97_4d20f337-ad17-4d2c-a2d3-e4e7c15b162d/dns-node-resolver/0.log"
Sep 17 14:08:51.765866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:51.765795    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:51.902006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:51.901974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:52.000482 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:52.000443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:52.000871 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:52.000489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:08:52.000871 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:52.000605    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:09:00.000583692 +0000 UTC m=+126.934088894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : configmap references non-existent config key: service-ca.crt
Sep 17 14:08:52.000871 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:52.000658    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 17 14:08:52.000871 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:52.000714    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs podName:183850d9-742f-4556-966d-99d1df6cc9a1 nodeName:}" failed. No retries permitted until 2026-09-17 14:09:00.000701312 +0000 UTC m=+126.934206494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs") pod "router-default-7d847c5fc8-txhzg" (UID: "183850d9-742f-4556-966d-99d1df6cc9a1") : secret "router-metrics-certs-default" not found
Sep 17 14:08:52.232149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:52.232124    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hztvs_f2312773-bed0-40d9-a40f-1b8f822d4c98/node-ca/0.log"
Sep 17 14:08:52.628598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:52.628569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:52.767469 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:52.767447    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:52.831423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:52.831396    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7wvpb_3d84b0d1-637b-439f-a466-1d55f375e01f/serve-healthcheck-canary/0.log"
Sep 17 14:08:52.869376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:52.869351    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:52.970885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:52.970823    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:53.008234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:53.008209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:53.008590 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:53.008277    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:53.008590 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:53.008346    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:53.008590 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:53.008357    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:53.008590 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:53.008393    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:08:57.008380503 +0000 UTC m=+123.941885672 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:53.008590 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:53.008412    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:08:57.008399112 +0000 UTC m=+123.941904282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:53.074105 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:53.074070    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:53.231998 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:53.231936    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:53.331243 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:53.331217    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:54.401214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:54.401175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:54.625182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:54.625156    2565 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:54.625367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:54.625204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:08:54.625612 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:54.625593    2565 scope.go:117] "RemoveContainer" containerID="c768691d2f590a099b2f1d34707e165e78da87977c1df2181d78701e594cdd0b"
Sep 17 14:08:54.625804 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:54.625781    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-dvbhf_openshift-console-operator(73c3fc4e-356f-49f8-a20d-18abf2d5c52a)\"" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf" podUID="73c3fc4e-356f-49f8-a20d-18abf2d5c52a"
Sep 17 14:08:54.762783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:54.762721    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:54.865014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:54.864984    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:54.988127 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:54.988102    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:55.153793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:55.153766    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:55.533396 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:55.533317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:56.176019 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:56.175992    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:56.547428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:56.547369    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:56.654803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:56.654775    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:56.815871 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:56.815817    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:56.986439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:56.986398    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:57.041889 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:57.041866    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:57.041989 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:57.041930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:08:57.042035 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:57.041981    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 14:08:57.042069 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:57.042035    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:57.042069 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:57.042045    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:09:05.042028547 +0000 UTC m=+131.975533719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : configmap "kube-rbac-proxy" not found
Sep 17 14:08:57.042138 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:08:57.042075    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls podName:85f6b33e-7505-4478-907f-f75fa8d8b07a nodeName:}" failed. No retries permitted until 2026-09-17 14:09:05.042062587 +0000 UTC m=+131.975567756 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mtkwv" (UID: "85f6b33e-7505-4478-907f-f75fa8d8b07a") : secret "insights-runtime-extractor-tls" not found
Sep 17 14:08:57.305800 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:57.305774    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:57.461952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:57.461930    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:59.879750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:59.879712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:08:59.977934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:08:59.977884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:00.061115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.061087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:00.061225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.061121    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:00.061762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.061737    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/183850d9-742f-4556-966d-99d1df6cc9a1-service-ca-bundle\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:00.063560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.063534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/183850d9-742f-4556-966d-99d1df6cc9a1-metrics-certs\") pod \"router-default-7d847c5fc8-txhzg\" (UID: \"183850d9-742f-4556-966d-99d1df6cc9a1\") " pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:00.125526 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.125500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:00.232867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.232802    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hc8ht\""
Sep 17 14:09:00.240951 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.240927    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:00.529801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.529734    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7d847c5fc8-txhzg"]
Sep 17 14:09:00.533587 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:00.533559    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183850d9_742f_4556_966d_99d1df6cc9a1.slice/crio-ae91139fef02ed5df35347f0c74fd94d94c2fea27abf169655aa1cb9a3411166 WatchSource:0}: Error finding container ae91139fef02ed5df35347f0c74fd94d94c2fea27abf169655aa1cb9a3411166: Status 404 returned error can't find the container with id ae91139fef02ed5df35347f0c74fd94d94c2fea27abf169655aa1cb9a3411166
Sep 17 14:09:00.735201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.735164    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:00.920591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:00.920562    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:01.025426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:01.025397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d847c5fc8-txhzg" event={"ID":"183850d9-742f-4556-966d-99d1df6cc9a1","Type":"ContainerStarted","Data":"23f3748368a27b5d66135662df9d1494fccee8d20fccd36bbb11e4e1bd28d4bb"}
Sep 17 14:09:01.025554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:01.025451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7d847c5fc8-txhzg" event={"ID":"183850d9-742f-4556-966d-99d1df6cc9a1","Type":"ContainerStarted","Data":"ae91139fef02ed5df35347f0c74fd94d94c2fea27abf169655aa1cb9a3411166"}
Sep 17 14:09:01.049247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:01.049195    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7d847c5fc8-txhzg" podStartSLOduration=17.049175485 podStartE2EDuration="17.049175485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:01.048524442 +0000 UTC m=+127.982029635" watchObservedRunningTime="2026-09-17 14:09:01.049175485 +0000 UTC m=+127.982680677"
Sep 17 14:09:01.124329 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:01.124301    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:01.241990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:01.241927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:01.244648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:01.244622    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:01.328093 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:01.328061    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:01.835337 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:01.835308    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:02.028604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.028578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:02.029888 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.029863    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7d847c5fc8-txhzg"
Sep 17 14:09:02.279561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.279500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:09:02.282085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.282054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d-metrics-certs\") pod \"network-metrics-daemon-h5mnt\" (UID: \"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d\") " pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:09:02.341270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.341228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:02.525759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.525728    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:02.537217 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.537165    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ck6nq\""
Sep 17 14:09:02.545193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.545168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-h5mnt"
Sep 17 14:09:02.737808 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.737776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:02.818017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.817957    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-h5mnt"]
Sep 17 14:09:02.820843 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:02.820819    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ffcc4c_e15b_4cc6_b0e6_f1c356cb2b3d.slice/crio-4bff887cb37f32e02005d17a474b46f2452f7344ada9232eeb4012c7a3d95c4c WatchSource:0}: Error finding container 4bff887cb37f32e02005d17a474b46f2452f7344ada9232eeb4012c7a3d95c4c: Status 404 returned error can't find the container with id 4bff887cb37f32e02005d17a474b46f2452f7344ada9232eeb4012c7a3d95c4c
Sep 17 14:09:02.952084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:02.952058    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:03.032751 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:03.032715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5mnt" event={"ID":"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d","Type":"ContainerStarted","Data":"4bff887cb37f32e02005d17a474b46f2452f7344ada9232eeb4012c7a3d95c4c"}
Sep 17 14:09:03.329654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:03.329625    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:03.817355 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:03.817332    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:04.037778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:04.037707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5mnt" event={"ID":"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d","Type":"ContainerStarted","Data":"6ad508d902774a7e4cc29d1ba2c7669e14079ff70c5049b7c212c55f81b2b76e"}
Sep 17 14:09:04.038166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:04.037789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-h5mnt" event={"ID":"e7ffcc4c-e15b-4cc6-b0e6-f1c356cb2b3d","Type":"ContainerStarted","Data":"e7ae97b9c8e3758464442bfea7dedae6ffd2a13552c883ea873fe5a4a15eaa12"}
Sep 17 14:09:04.315465 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:04.315394    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:04.415762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:04.415730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:04.512910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:04.512882    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:04.770490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:04.770455    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:05.100831 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:05.100804    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:09:05.101200 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:05.100864    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:09:05.101524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:05.101502    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85f6b33e-7505-4478-907f-f75fa8d8b07a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:09:05.103246 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:05.103223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85f6b33e-7505-4478-907f-f75fa8d8b07a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mtkwv\" (UID: \"85f6b33e-7505-4478-907f-f75fa8d8b07a\") " pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:09:05.257908 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:05.257876    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmmrx\""
Sep 17 14:09:05.265827 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:05.265806    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mtkwv"
Sep 17 14:09:05.541801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:05.541691    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-h5mnt" podStartSLOduration=130.626859711 podStartE2EDuration="2m11.541641869s" totalImagesPullingTime="914.782158ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:06:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:04.056565773 +0000 UTC m=+130.990070964" watchObservedRunningTime="2026-09-17 14:09:05.541641869 +0000 UTC m=+132.475147061"
Sep 17 14:09:05.543051 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:05.543027    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mtkwv"]
Sep 17 14:09:05.545558 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:05.545533    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f6b33e_7505_4478_907f_f75fa8d8b07a.slice/crio-3af5ec2d25efeab61dec4b9c46529c17b93e16b127f7117548c3e682c9516db3 WatchSource:0}: Error finding container 3af5ec2d25efeab61dec4b9c46529c17b93e16b127f7117548c3e682c9516db3: Status 404 returned error can't find the container with id 3af5ec2d25efeab61dec4b9c46529c17b93e16b127f7117548c3e682c9516db3
Sep 17 14:09:06.046083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:06.046047    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtkwv" event={"ID":"85f6b33e-7505-4478-907f-f75fa8d8b07a","Type":"ContainerStarted","Data":"29038281bd7837ae05f2b732d84bdf5816ad2a74dc04a832b297d48c42ccdc3a"}
Sep 17 14:09:06.046229 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:06.046109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtkwv" event={"ID":"85f6b33e-7505-4478-907f-f75fa8d8b07a","Type":"ContainerStarted","Data":"3af5ec2d25efeab61dec4b9c46529c17b93e16b127f7117548c3e682c9516db3"}
Sep 17 14:09:06.513918 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:06.513895    2565 scope.go:117] "RemoveContainer" containerID="c768691d2f590a099b2f1d34707e165e78da87977c1df2181d78701e594cdd0b"
Sep 17 14:09:06.853364 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:06.853322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:06.963081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:06.963049    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:07.053699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.053674    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:09:07.053957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.053935    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf" event={"ID":"73c3fc4e-356f-49f8-a20d-18abf2d5c52a","Type":"ContainerStarted","Data":"d34018242d82a90201bb1a9aff3b6f5f37b44d595b4ee14402722e45e9bb7281"}
Sep 17 14:09:07.054955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.054937    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:09:07.060335 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.060316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf"
Sep 17 14:09:07.085382 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.085341    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-dvbhf" podStartSLOduration=19.991593933 podStartE2EDuration="23.085328179s" totalImagesPullingTime="3.093734246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:08:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:07.084895877 +0000 UTC m=+134.018401069" watchObservedRunningTime="2026-09-17 14:09:07.085328179 +0000 UTC m=+134.018833374"
Sep 17 14:09:07.108540 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.108464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:07.201271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.201225    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:07.385496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.385414    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:07.603025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.602993    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:07.722486 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.722419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:07.833272 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:07.833234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:08.042388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:08.042297    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:08.216174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:08.216145    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:09.064326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:09.063863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtkwv" event={"ID":"85f6b33e-7505-4478-907f-f75fa8d8b07a","Type":"ContainerStarted","Data":"7a8663f0cedfab10c66d0da9f162e04d5431e2da73a539f95a65eef2d71e03a8"}
Sep 17 14:09:09.068762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:09.068690    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:09.319404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:09.319326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:09.516300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:09.516262    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:10.069067 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:10.069034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mtkwv" event={"ID":"85f6b33e-7505-4478-907f-f75fa8d8b07a","Type":"ContainerStarted","Data":"3f5f5c7b39224cdd4be7ffdc8d211b4171387faef5394baab0c14aeaa96f0c7c"}
Sep 17 14:09:10.090478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:10.090416    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mtkwv" podStartSLOduration=17.357298465 podStartE2EDuration="21.090396307s" totalImagesPullingTime="3.733097842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:08:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:10.089957393 +0000 UTC m=+137.023462585" watchObservedRunningTime="2026-09-17 14:09:10.090396307 +0000 UTC m=+137.023901499"
Sep 17 14:09:10.214645 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:10.214618    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:10.284228 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:10.284196    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:15.508043 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.508002    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"]
Sep 17 14:09:15.511455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.511428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"
Sep 17 14:09:15.516680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.516652    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 14:09:15.516680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.516670    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-86h8t\""
Sep 17 14:09:15.516906 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.516651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 14:09:15.528978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.528953    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"]
Sep 17 14:09:15.574464 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.574440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f896ec3f-2d7c-46d4-978d-c87c9187a60b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xzvqn\" (UID: \"f896ec3f-2d7c-46d4-978d-c87c9187a60b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"
Sep 17 14:09:15.574589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.574576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f896ec3f-2d7c-46d4-978d-c87c9187a60b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xzvqn\" (UID: \"f896ec3f-2d7c-46d4-978d-c87c9187a60b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"
Sep 17 14:09:15.628712 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.628690    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"]
Sep 17 14:09:15.631473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.631459    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"
Sep 17 14:09:15.635508 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.635491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 14:09:15.643032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.643014    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gggcq\""
Sep 17 14:09:15.670289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.670266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"]
Sep 17 14:09:15.675450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.675427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f896ec3f-2d7c-46d4-978d-c87c9187a60b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xzvqn\" (UID: \"f896ec3f-2d7c-46d4-978d-c87c9187a60b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"
Sep 17 14:09:15.675539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.675462    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f896ec3f-2d7c-46d4-978d-c87c9187a60b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xzvqn\" (UID: \"f896ec3f-2d7c-46d4-978d-c87c9187a60b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"
Sep 17 14:09:15.675968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.675952    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f896ec3f-2d7c-46d4-978d-c87c9187a60b-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-xzvqn\" (UID: \"f896ec3f-2d7c-46d4-978d-c87c9187a60b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"
Sep 17 14:09:15.677766 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.677744    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f896ec3f-2d7c-46d4-978d-c87c9187a60b-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-xzvqn\" (UID: \"f896ec3f-2d7c-46d4-978d-c87c9187a60b\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"
Sep 17 14:09:15.728032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.727997    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:15.775890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.775833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/63d3b0a0-ef56-4182-9395-ed221f69ee85-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-z2d6q\" (UID: \"63d3b0a0-ef56-4182-9395-ed221f69ee85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"
Sep 17 14:09:15.821668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.821644    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"
Sep 17 14:09:15.822820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.822799    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-86df9b5656-bq6mt"]
Sep 17 14:09:15.825546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.825527    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.829464 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.829443    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 14:09:15.829566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.829493    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 14:09:15.829654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.829638    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 14:09:15.830690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.830669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2g4kt\""
Sep 17 14:09:15.835945 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.835924    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 14:09:15.850060 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.850035    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86df9b5656-bq6mt"]
Sep 17 14:09:15.876583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.876562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/63d3b0a0-ef56-4182-9395-ed221f69ee85-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-z2d6q\" (UID: \"63d3b0a0-ef56-4182-9395-ed221f69ee85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"
Sep 17 14:09:15.877495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.877475    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:15.879189 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.879169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/63d3b0a0-ef56-4182-9395-ed221f69ee85-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-z2d6q\" (UID: \"63d3b0a0-ef56-4182-9395-ed221f69ee85\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"
Sep 17 14:09:15.939235 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.939208    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"
Sep 17 14:09:15.977298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.977270    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51870341-f4ff-4804-98c3-31864864cc92-image-registry-private-configuration\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.977400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.977307    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51870341-f4ff-4804-98c3-31864864cc92-installation-pull-secrets\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.977400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.977331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxxh\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-kube-api-access-wpxxh\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.977400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.977351    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51870341-f4ff-4804-98c3-31864864cc92-trusted-ca\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.977400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.977370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51870341-f4ff-4804-98c3-31864864cc92-registry-certificates\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.977548 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.977432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-registry-tls\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.977548 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.977457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-bound-sa-token\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.977548 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.977494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51870341-f4ff-4804-98c3-31864864cc92-ca-trust-extracted\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:15.987534 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:15.987443    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:16.078045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078021    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-bound-sa-token\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.078140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51870341-f4ff-4804-98c3-31864864cc92-ca-trust-extracted\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.078140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51870341-f4ff-4804-98c3-31864864cc92-image-registry-private-configuration\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.078140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51870341-f4ff-4804-98c3-31864864cc92-installation-pull-secrets\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.078317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxxh\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-kube-api-access-wpxxh\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.078317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51870341-f4ff-4804-98c3-31864864cc92-trusted-ca\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.078317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51870341-f4ff-4804-98c3-31864864cc92-registry-certificates\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.078460 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-registry-tls\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.078514 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.078470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/51870341-f4ff-4804-98c3-31864864cc92-ca-trust-extracted\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.079192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.079164    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/51870341-f4ff-4804-98c3-31864864cc92-registry-certificates\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.079319 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.079192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/51870341-f4ff-4804-98c3-31864864cc92-trusted-ca\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.080484 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.080456    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/51870341-f4ff-4804-98c3-31864864cc92-installation-pull-secrets\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.080584 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.080543    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/51870341-f4ff-4804-98c3-31864864cc92-image-registry-private-configuration\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.080667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.080648    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-registry-tls\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.089397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.089374    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-bound-sa-token\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.089630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.089616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxxh\" (UniqueName: \"kubernetes.io/projected/51870341-f4ff-4804-98c3-31864864cc92-kube-api-access-wpxxh\") pod \"image-registry-86df9b5656-bq6mt\" (UID: \"51870341-f4ff-4804-98c3-31864864cc92\") " pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.101320 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.101300    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn"]
Sep 17 14:09:16.103948 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:16.103926    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf896ec3f_2d7c_46d4_978d_c87c9187a60b.slice/crio-00720658a5e17acf56ec6f6334a78acb5c5cda37c2dc42fef2c7542f35f915ec WatchSource:0}: Error finding container 00720658a5e17acf56ec6f6334a78acb5c5cda37c2dc42fef2c7542f35f915ec: Status 404 returned error can't find the container with id 00720658a5e17acf56ec6f6334a78acb5c5cda37c2dc42fef2c7542f35f915ec
Sep 17 14:09:16.143608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.143587    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:16.206699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.206668    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"]
Sep 17 14:09:16.210415 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:16.210389    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d3b0a0_ef56_4182_9395_ed221f69ee85.slice/crio-49929b072843dac7e0dd804c453d73b91d8fdee38da3961bcb009c8ecde1b424 WatchSource:0}: Error finding container 49929b072843dac7e0dd804c453d73b91d8fdee38da3961bcb009c8ecde1b424: Status 404 returned error can't find the container with id 49929b072843dac7e0dd804c453d73b91d8fdee38da3961bcb009c8ecde1b424
Sep 17 14:09:16.419547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.419528    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-86df9b5656-bq6mt"]
Sep 17 14:09:16.421241 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:16.421215    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51870341_f4ff_4804_98c3_31864864cc92.slice/crio-6f30ad09fe9c10715e676a7f339dddad3582a51da216afe04f299a8859f05eaa WatchSource:0}: Error finding container 6f30ad09fe9c10715e676a7f339dddad3582a51da216afe04f299a8859f05eaa: Status 404 returned error can't find the container with id 6f30ad09fe9c10715e676a7f339dddad3582a51da216afe04f299a8859f05eaa
Sep 17 14:09:16.626245 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.626205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:16.733667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.733590    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:16.849522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.849488    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59c9ffcc65-7btds"]
Sep 17 14:09:16.854595 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.854408    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:16.858646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.858620    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 14:09:16.858774 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.858684    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 14:09:16.858774 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.858748    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 14:09:16.858917 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.858858    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 14:09:16.859017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.858997    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-45dp8\""
Sep 17 14:09:16.859139 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.859117    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 14:09:16.859827 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.859757    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 14:09:16.861315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.861102    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 14:09:16.873302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.871824    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c9ffcc65-7btds"]
Sep 17 14:09:16.987557 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.987488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-oauth-serving-cert\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:16.987557 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.987524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-config\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:16.987736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.987609    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-serving-cert\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:16.987736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.987641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-service-ca\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:16.987736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.987675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7bbn\" (UniqueName: \"kubernetes.io/projected/48aefd83-e480-4b9e-bae8-92d9c88798bf-kube-api-access-c7bbn\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:16.987843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:16.987781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-oauth-config\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.088146 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.088115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-oauth-config\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.088331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.088160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-oauth-serving-cert\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.088331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.088187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-config\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.088331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.088266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-serving-cert\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.088331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.088298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-service-ca\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.088550 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.088330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7bbn\" (UniqueName: \"kubernetes.io/projected/48aefd83-e480-4b9e-bae8-92d9c88798bf-kube-api-access-c7bbn\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.088944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.088920    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-oauth-serving-cert\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.089069 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.089000    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-config\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.089131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.089061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-service-ca\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.090889 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.090868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-oauth-config\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.091161 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.091137    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-serving-cert\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.093661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.093623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q" event={"ID":"63d3b0a0-ef56-4182-9395-ed221f69ee85","Type":"ContainerStarted","Data":"49929b072843dac7e0dd804c453d73b91d8fdee38da3961bcb009c8ecde1b424"}
Sep 17 14:09:17.095051 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.095026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn" event={"ID":"f896ec3f-2d7c-46d4-978d-c87c9187a60b","Type":"ContainerStarted","Data":"00720658a5e17acf56ec6f6334a78acb5c5cda37c2dc42fef2c7542f35f915ec"}
Sep 17 14:09:17.096673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.096650    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86df9b5656-bq6mt" event={"ID":"51870341-f4ff-4804-98c3-31864864cc92","Type":"ContainerStarted","Data":"5546f9a9e7f403c61462d0f06480c807a776818781313be406390b97623fad22"}
Sep 17 14:09:17.096776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.096694    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-86df9b5656-bq6mt" event={"ID":"51870341-f4ff-4804-98c3-31864864cc92","Type":"ContainerStarted","Data":"6f30ad09fe9c10715e676a7f339dddad3582a51da216afe04f299a8859f05eaa"}
Sep 17 14:09:17.096838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.096822    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:17.100496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.100474    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7bbn\" (UniqueName: \"kubernetes.io/projected/48aefd83-e480-4b9e-bae8-92d9c88798bf-kube-api-access-c7bbn\") pod \"console-59c9ffcc65-7btds\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") " pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.166610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.166585    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:17.824141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.824079    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-86df9b5656-bq6mt" podStartSLOduration=2.824057505 podStartE2EDuration="2.824057505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:17.124350019 +0000 UTC m=+144.057855225" watchObservedRunningTime="2026-09-17 14:09:17.824057505 +0000 UTC m=+144.757562695"
Sep 17 14:09:17.825380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:17.825359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c9ffcc65-7btds"]
Sep 17 14:09:17.827766 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:17.827738    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48aefd83_e480_4b9e_bae8_92d9c88798bf.slice/crio-01d64a803b4313622831b7c8cda37b1ac58e0fe73db8b416d27178060dcce9b6 WatchSource:0}: Error finding container 01d64a803b4313622831b7c8cda37b1ac58e0fe73db8b416d27178060dcce9b6: Status 404 returned error can't find the container with id 01d64a803b4313622831b7c8cda37b1ac58e0fe73db8b416d27178060dcce9b6
Sep 17 14:09:18.102617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.102530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q" event={"ID":"63d3b0a0-ef56-4182-9395-ed221f69ee85","Type":"ContainerStarted","Data":"e9cf65298301399b78dfe5974eebe191e31f0620bf5f18e01b6883516bbd539e"}
Sep 17 14:09:18.102617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.102603    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"
Sep 17 14:09:18.104611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.104576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn" event={"ID":"f896ec3f-2d7c-46d4-978d-c87c9187a60b","Type":"ContainerStarted","Data":"8b758defc31a0c57abc3cc25941bac59a5538922ed381f858b639a035431bb20"}
Sep 17 14:09:18.106320 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.106291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c9ffcc65-7btds" event={"ID":"48aefd83-e480-4b9e-bae8-92d9c88798bf","Type":"ContainerStarted","Data":"01d64a803b4313622831b7c8cda37b1ac58e0fe73db8b416d27178060dcce9b6"}
Sep 17 14:09:18.108177 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.108154    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q"
Sep 17 14:09:18.120923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.120870    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-z2d6q" podStartSLOduration=1.779434866 podStartE2EDuration="3.120855235s" totalImagesPullingTime="1.341420369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:18.119212748 +0000 UTC m=+145.052717961" watchObservedRunningTime="2026-09-17 14:09:18.120855235 +0000 UTC m=+145.054360426"
Sep 17 14:09:18.145472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.145426    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-xzvqn" podStartSLOduration=1.6959698479999998 podStartE2EDuration="3.145412943s" totalImagesPullingTime="1.449443095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:18.141980527 +0000 UTC m=+145.075485720" watchObservedRunningTime="2026-09-17 14:09:18.145412943 +0000 UTC m=+145.078918179"
Sep 17 14:09:18.275144 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.275117    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:18.346153 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.346130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:18.541325 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.541235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:18.667158 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.667129    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:18.817506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.817421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:18.905661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:18.905633    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:19.000667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:19.000636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:20.883698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:20.883665    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:21.119751 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:21.119721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c9ffcc65-7btds" event={"ID":"48aefd83-e480-4b9e-bae8-92d9c88798bf","Type":"ContainerStarted","Data":"8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8"}
Sep 17 14:09:21.140749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:21.140682    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59c9ffcc65-7btds" podStartSLOduration=2.318634516 podStartE2EDuration="5.140644473s" totalImagesPullingTime="2.822009957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:21.139856075 +0000 UTC m=+148.073361268" watchObservedRunningTime="2026-09-17 14:09:21.140644473 +0000 UTC m=+148.074149665"
Sep 17 14:09:22.821134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:22.821098    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:23.994518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:23.994481    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-xp9xh"]
Sep 17 14:09:23.997410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:23.997391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.002821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:23.999706    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 14:09:24.002821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:23.999980    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rvp8v\""
Sep 17 14:09:24.002821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.000297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 14:09:24.002821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.000352    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 14:09:24.002821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.000995    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 14:09:24.002821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.001125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 14:09:24.002821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.001295    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 14:09:24.033108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.033083    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"]
Sep 17 14:09:24.035404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.035388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.037808 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.037787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mj9v9\""
Sep 17 14:09:24.038267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.038239    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 17 14:09:24.038362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.038263    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 17 14:09:24.038362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.038293    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 17 14:09:24.050924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.050902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"]
Sep 17 14:09:24.140529 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-root\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.140634 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfhpj\" (UniqueName: \"kubernetes.io/projected/e63b607b-6798-4045-be4a-1084d98b2b08-kube-api-access-jfhpj\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.140634 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-sys\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.140634 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffs5j\" (UniqueName: \"kubernetes.io/projected/bd6a4dfe-df50-416f-9718-022d6c974565-kube-api-access-ffs5j\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.140634 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e63b607b-6798-4045-be4a-1084d98b2b08-metrics-client-ca\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.140829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.140829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.140829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140745    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.140829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-tls\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.140829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-textfile\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.140982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd6a4dfe-df50-416f-9718-022d6c974565-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.140982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140873    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.140982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd6a4dfe-df50-416f-9718-022d6c974565-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.140982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-wtmp\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.140982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.140947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-accelerators-collector-config\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.241312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.241393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241319    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.241393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.241468 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:09:24.241420    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 14:09:24.241468 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:09:24.241467    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls podName:bd6a4dfe-df50-416f-9718-022d6c974565 nodeName:}" failed. No retries permitted until 2026-09-17 14:09:24.741452959 +0000 UTC m=+151.674958129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-nsmsq" (UID: "bd6a4dfe-df50-416f-9718-022d6c974565") : secret "kube-state-metrics-tls" not found
Sep 17 14:09:24.241568 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241482    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-tls\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.241568 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-textfile\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.241568 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd6a4dfe-df50-416f-9718-022d6c974565-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.241568 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.241778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd6a4dfe-df50-416f-9718-022d6c974565-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.241778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-wtmp\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.241778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-accelerators-collector-config\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.241778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241690    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-root\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.241778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241719    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfhpj\" (UniqueName: \"kubernetes.io/projected/e63b607b-6798-4045-be4a-1084d98b2b08-kube-api-access-jfhpj\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.241778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-sys\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.242131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241789    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffs5j\" (UniqueName: \"kubernetes.io/projected/bd6a4dfe-df50-416f-9718-022d6c974565-kube-api-access-ffs5j\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.242131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e63b607b-6798-4045-be4a-1084d98b2b08-metrics-client-ca\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.242131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241855    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-wtmp\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.242131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.241935    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-sys\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.242131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.242029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/bd6a4dfe-df50-416f-9718-022d6c974565-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.242410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.242203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e63b607b-6798-4045-be4a-1084d98b2b08-root\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.242410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.242376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-textfile\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.242691 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.242664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e63b607b-6798-4045-be4a-1084d98b2b08-metrics-client-ca\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.242780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.242760    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-accelerators-collector-config\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.242910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.242882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.242986 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.242959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd6a4dfe-df50-416f-9718-022d6c974565-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.244127 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.244097    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.244270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.244235    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.244345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.244328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e63b607b-6798-4045-be4a-1084d98b2b08-node-exporter-tls\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.255330 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.255273    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffs5j\" (UniqueName: \"kubernetes.io/projected/bd6a4dfe-df50-416f-9718-022d6c974565-kube-api-access-ffs5j\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.258139 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.258119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfhpj\" (UniqueName: \"kubernetes.io/projected/e63b607b-6798-4045-be4a-1084d98b2b08-kube-api-access-jfhpj\") pod \"node-exporter-xp9xh\" (UID: \"e63b607b-6798-4045-be4a-1084d98b2b08\") " pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.310203 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.310180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-xp9xh"
Sep 17 14:09:24.320617 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:24.320595    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63b607b_6798_4045_be4a_1084d98b2b08.slice/crio-75ed3dfd02cf6d1ac2c0865ce34742d2d8dbfdeccc10b5f7c298818cd11cb186 WatchSource:0}: Error finding container 75ed3dfd02cf6d1ac2c0865ce34742d2d8dbfdeccc10b5f7c298818cd11cb186: Status 404 returned error can't find the container with id 75ed3dfd02cf6d1ac2c0865ce34742d2d8dbfdeccc10b5f7c298818cd11cb186
Sep 17 14:09:24.745659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.745624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:24.745811 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:09:24.745768    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 17 14:09:24.745852 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:09:24.745836    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls podName:bd6a4dfe-df50-416f-9718-022d6c974565 nodeName:}" failed. No retries permitted until 2026-09-17 14:09:25.745820336 +0000 UTC m=+152.679325510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-nsmsq" (UID: "bd6a4dfe-df50-416f-9718-022d6c974565") : secret "kube-state-metrics-tls" not found
Sep 17 14:09:24.755234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.755207    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:24.855019 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:24.854993    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:25.072638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.072545    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:25.135156 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.135111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp9xh" event={"ID":"e63b607b-6798-4045-be4a-1084d98b2b08","Type":"ContainerStarted","Data":"75ed3dfd02cf6d1ac2c0865ce34742d2d8dbfdeccc10b5f7c298818cd11cb186"}
Sep 17 14:09:25.197535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.197494    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:09:25.201285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.201263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.203636 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.203610    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 17 14:09:25.203947 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.203774    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 17 14:09:25.203947 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.203802    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 17 14:09:25.203947 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.203822    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 17 14:09:25.203947 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.203948    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lc9pn\""
Sep 17 14:09:25.205129 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.205106    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 17 14:09:25.205129 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.205116    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 17 14:09:25.205316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.205116    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 17 14:09:25.205316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.205186    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 17 14:09:25.206191 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.206173    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 17 14:09:25.212207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.212181    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:25.238733 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.238710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:09:25.352565 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-web-config\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352623    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-out\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352805    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352843    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352879    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352899    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9ld\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-kube-api-access-nt9ld\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.352930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352929    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.353155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.352964    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.404607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.404584    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:25.453556 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453535    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-web-config\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453782    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-out\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.453927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.454169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.454169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.453964    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9ld\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-kube-api-access-nt9ld\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.454169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.454052    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.454967 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.454908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.455934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.455888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.456246 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.456220    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.456401 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.456357    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-web-config\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.456589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.456571    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.456820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.456799    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.456870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.456824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-out\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.456951 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.456926    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.457198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.457175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.457957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.457938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.458104 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.458090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.473419 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.473400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9ld\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-kube-api-access-nt9ld\") pod \"alertmanager-main-0\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.513389 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.513368    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:09:25.757404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.757328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:25.759623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.759603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/bd6a4dfe-df50-416f-9718-022d6c974565-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-nsmsq\" (UID: \"bd6a4dfe-df50-416f-9718-022d6c974565\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:25.803815 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.803797    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:09:25.806733 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:25.806704    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc10f32b4_c9b2_406f_b88c_a45e9e43e8a6.slice/crio-91dcded040b14c52548eaa662f87675c677b0d4567cf7110fc9cc8421274de09 WatchSource:0}: Error finding container 91dcded040b14c52548eaa662f87675c677b0d4567cf7110fc9cc8421274de09: Status 404 returned error can't find the container with id 91dcded040b14c52548eaa662f87675c677b0d4567cf7110fc9cc8421274de09
Sep 17 14:09:25.844224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:25.844203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"
Sep 17 14:09:26.019866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.019806    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-85c49449dc-gqzzd"]
Sep 17 14:09:26.022829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.022809    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.027694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.027669    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 17 14:09:26.027843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.027827    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 17 14:09:26.027915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.027893    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 17 14:09:26.027978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.027958    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 17 14:09:26.028110 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.028089    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 17 14:09:26.031797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.031777    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j66zc\""
Sep 17 14:09:26.031896 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.031819    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9dniq0sl7d1a1\""
Sep 17 14:09:26.044051 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.044032    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85c49449dc-gqzzd"]
Sep 17 14:09:26.116712 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.116672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq"]
Sep 17 14:09:26.119034 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:26.119014    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6a4dfe_df50_416f_9718_022d6c974565.slice/crio-0f075eb3aaf26089e86561a63523526ae4c54a22ba733b74931988ee4184ac03 WatchSource:0}: Error finding container 0f075eb3aaf26089e86561a63523526ae4c54a22ba733b74931988ee4184ac03: Status 404 returned error can't find the container with id 0f075eb3aaf26089e86561a63523526ae4c54a22ba733b74931988ee4184ac03
Sep 17 14:09:26.140043 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.140016    2565 generic.go:356] "Generic (PLEG): container finished" podID="e63b607b-6798-4045-be4a-1084d98b2b08" containerID="c70ff10043b40da26595dc1c93ae45ecaefd8fbe9f0660ee0ca52d538c58ccc4" exitCode=0
Sep 17 14:09:26.140176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.140104    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp9xh" event={"ID":"e63b607b-6798-4045-be4a-1084d98b2b08","Type":"ContainerDied","Data":"c70ff10043b40da26595dc1c93ae45ecaefd8fbe9f0660ee0ca52d538c58ccc4"}
Sep 17 14:09:26.141410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.141389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq" event={"ID":"bd6a4dfe-df50-416f-9718-022d6c974565","Type":"ContainerStarted","Data":"0f075eb3aaf26089e86561a63523526ae4c54a22ba733b74931988ee4184ac03"}
Sep 17 14:09:26.142486 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.142466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerStarted","Data":"91dcded040b14c52548eaa662f87675c677b0d4567cf7110fc9cc8421274de09"}
Sep 17 14:09:26.160560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.160539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.160651 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.160594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.160690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.160656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-tls\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.160725 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.160692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.160755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.160719    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.160805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.160771    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-grpc-tls\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.160844 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.160801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9v86\" (UniqueName: \"kubernetes.io/projected/aa638175-c8b8-4ae6-8769-31d4ba1b0395-kube-api-access-l9v86\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.160880 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.160840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa638175-c8b8-4ae6-8769-31d4ba1b0395-metrics-client-ca\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.261503 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.261480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.261590 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.261542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-tls\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.261699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.261673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.261753 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.261729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.261806 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.261778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-grpc-tls\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.261861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.261805    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9v86\" (UniqueName: \"kubernetes.io/projected/aa638175-c8b8-4ae6-8769-31d4ba1b0395-kube-api-access-l9v86\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.261861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.261844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa638175-c8b8-4ae6-8769-31d4ba1b0395-metrics-client-ca\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.261962 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.261946    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.262704 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.262678    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa638175-c8b8-4ae6-8769-31d4ba1b0395-metrics-client-ca\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.264436 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.264410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.264688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.264667    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.264776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.264715    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.264898 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.264874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-tls\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.264936 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.264918    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-grpc-tls\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.265052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.265032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa638175-c8b8-4ae6-8769-31d4ba1b0395-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.270742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.270691    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9v86\" (UniqueName: \"kubernetes.io/projected/aa638175-c8b8-4ae6-8769-31d4ba1b0395-kube-api-access-l9v86\") pod \"thanos-querier-85c49449dc-gqzzd\" (UID: \"aa638175-c8b8-4ae6-8769-31d4ba1b0395\") " pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.331688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.331661    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:26.612587 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.612553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-85c49449dc-gqzzd"]
Sep 17 14:09:26.617192 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:26.617160    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa638175_c8b8_4ae6_8769_31d4ba1b0395.slice/crio-a86ef1a8feea6172487970f2b02b5faa96f7a9a322de52cf7434d9c993b9f4bc WatchSource:0}: Error finding container a86ef1a8feea6172487970f2b02b5faa96f7a9a322de52cf7434d9c993b9f4bc: Status 404 returned error can't find the container with id a86ef1a8feea6172487970f2b02b5faa96f7a9a322de52cf7434d9c993b9f4bc
Sep 17 14:09:26.996722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:26.996648    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:27.102605 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.102571    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:27.148957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.148914    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd" event={"ID":"aa638175-c8b8-4ae6-8769-31d4ba1b0395","Type":"ContainerStarted","Data":"a86ef1a8feea6172487970f2b02b5faa96f7a9a322de52cf7434d9c993b9f4bc"}
Sep 17 14:09:27.151539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.151498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp9xh" event={"ID":"e63b607b-6798-4045-be4a-1084d98b2b08","Type":"ContainerStarted","Data":"b3f7933459eed9a8fb372c85dc0ef09fc085592f803ffb03a50b299d2952a735"}
Sep 17 14:09:27.151664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.151550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-xp9xh" event={"ID":"e63b607b-6798-4045-be4a-1084d98b2b08","Type":"ContainerStarted","Data":"f92e318c310b30eeb11a6df091ba8ed26d40d5a442c940d9cec87e542a2317b8"}
Sep 17 14:09:27.167553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.167527    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:27.167654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.167590    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:27.169003 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.168976    2565 patch_prober.go:28] interesting pod/console-59c9ffcc65-7btds container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.17:8443/health\": dial tcp 10.134.0.17:8443: connect: connection refused" start-of-body=
Sep 17 14:09:27.169088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.169043    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-59c9ffcc65-7btds" podUID="48aefd83-e480-4b9e-bae8-92d9c88798bf" containerName="console" probeResult="failure" output="Get \"https://10.134.0.17:8443/health\": dial tcp 10.134.0.17:8443: connect: connection refused"
Sep 17 14:09:27.239665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.239640    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:27.385687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.385663    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:27.514420 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:27.514382    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:28.158599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:28.158419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq" event={"ID":"bd6a4dfe-df50-416f-9718-022d6c974565","Type":"ContainerStarted","Data":"d0be0add3de2254edbbb04fcb4486103985a6f07a012bbfd10ded42b88ea55db"}
Sep 17 14:09:28.158599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:28.158516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq" event={"ID":"bd6a4dfe-df50-416f-9718-022d6c974565","Type":"ContainerStarted","Data":"71b28d553e99d85534c67a35898179b8051eaba36aa969391eeefab9691654af"}
Sep 17 14:09:28.158599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:28.158548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq" event={"ID":"bd6a4dfe-df50-416f-9718-022d6c974565","Type":"ContainerStarted","Data":"e8a006169e98d52652349d0c0f6451c16fcf39a5a7ec02d2b39c85fec1ef4a89"}
Sep 17 14:09:28.207846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:28.207788    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-xp9xh" podStartSLOduration=4.1780447 podStartE2EDuration="5.207765709s" totalImagesPullingTime="968.396176ms" totalInitContainerRuntime="61.324833ms" isStatefulPod=true podCreationTimestamp="2026-09-17 14:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:27.180524907 +0000 UTC m=+154.114030095" watchObservedRunningTime="2026-09-17 14:09:28.207765709 +0000 UTC m=+155.141270902"
Sep 17 14:09:28.209125 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:28.209068    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-nsmsq" podStartSLOduration=3.118031254 podStartE2EDuration="4.209055397s" totalImagesPullingTime="1.091024143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:28.205813888 +0000 UTC m=+155.139319082" watchObservedRunningTime="2026-09-17 14:09:28.209055397 +0000 UTC m=+155.142560595"
Sep 17 14:09:28.631296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:28.631240    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:28.823669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:28.823634    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:28.924338 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:09:28.924246    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-flvcb" podUID="3b860267-277a-4b1b-85b8-850e86d2cb3d"
Sep 17 14:09:28.932502 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:28.932472    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:29.158969 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:29.158941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:29.163923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:29.163897    2565 generic.go:356] "Generic (PLEG): container finished" podID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerID="44ae9a325b3f1fd876a61423f8e11a9ffcd0bf8ee1bdd12cfdf76c9cbde19b8f" exitCode=0
Sep 17 14:09:29.164235 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:29.163983    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerDied","Data":"44ae9a325b3f1fd876a61423f8e11a9ffcd0bf8ee1bdd12cfdf76c9cbde19b8f"}
Sep 17 14:09:29.166450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:29.166428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd" event={"ID":"aa638175-c8b8-4ae6-8769-31d4ba1b0395","Type":"ContainerStarted","Data":"d80f247990ec84b1d457293936d023a19d741e2bf15cea80b850674b94cb5ad4"}
Sep 17 14:09:29.166580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:29.166456    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-flvcb"
Sep 17 14:09:29.166580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:29.166482    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd" event={"ID":"aa638175-c8b8-4ae6-8769-31d4ba1b0395","Type":"ContainerStarted","Data":"f534ab9c73c31ac8df7ea60f4073ad0d4c7855c9aa46c51d8d324d1e52f4848f"}
Sep 17 14:09:29.166580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:29.166511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd" event={"ID":"aa638175-c8b8-4ae6-8769-31d4ba1b0395","Type":"ContainerStarted","Data":"0a74d2e7c0d6bb85abc67975bfba19f8f46deafc06c14b34dd83a37bdfebf8cf"}
Sep 17 14:09:30.295144 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.295068    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:30.300917 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.300887    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:09:30.303979 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.303955    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.308167 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.308131    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 14:09:30.308318 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.308131    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 14:09:30.308318 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.308214    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 14:09:30.308446 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.308388    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 14:09:30.308638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.308617    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 14:09:30.309076 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.309046    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 14:09:30.309594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.309369    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9jefi79h9ugaa\""
Sep 17 14:09:30.309594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.309409    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 14:09:30.309594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.309426    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 14:09:30.309594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.309441    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 14:09:30.309594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.309469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v9bsz\""
Sep 17 14:09:30.309594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.309496    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 14:09:30.309594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.309561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 14:09:30.310066 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.310037    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 14:09:30.310150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.310137    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 14:09:30.311534 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.311506    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 14:09:30.318995 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.318974    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 14:09:30.326684 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.326659    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:09:30.392188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392219    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-web-config\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392655 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392530    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-config-out\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392655 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392655 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392655 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392655 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh5wb\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-kube-api-access-bh5wb\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392891 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392891 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.392891 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.393014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.393014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.392951    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.393014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.393002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.393141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.393032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-config\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.393141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.393053    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494235 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494235 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494235 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494444 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-config-out\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494492 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494542 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh5wb\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-kube-api-access-bh5wb\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-config\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.494846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.495084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494861    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.495084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.495084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494907    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-web-config\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.495084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.494945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.495459 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.495434    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.495644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.495617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.495989 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.495958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.496610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.496582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.498097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.497720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.498097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.498046    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-web-config\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.498097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.498088    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.498322 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.498264    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.498495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.498469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-config-out\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.498615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.498592    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.498999 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.498964    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.499131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.499108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.499361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.499332    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-config\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.499631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.499606    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.499857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.499830    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.500217 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.500196    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.500731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.500709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.501072 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.501054    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.503818 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.503788    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh5wb\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-kube-api-access-bh5wb\") pod \"prometheus-k8s-0\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.614696 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.614673    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:30.899320 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:30.899289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:09:30.901739 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:30.901710    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66999a84_0b96_49ca_8a59_6ffc91f76963.slice/crio-8235b46db1667e8e9c8c5f715f6771776de0383c6af57159d3e32f2edcea8d69 WatchSource:0}: Error finding container 8235b46db1667e8e9c8c5f715f6771776de0383c6af57159d3e32f2edcea8d69: Status 404 returned error can't find the container with id 8235b46db1667e8e9c8c5f715f6771776de0383c6af57159d3e32f2edcea8d69
Sep 17 14:09:31.176722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:31.176657    2565 generic.go:356] "Generic (PLEG): container finished" podID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerID="8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e" exitCode=0
Sep 17 14:09:31.176818 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:31.176748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerDied","Data":"8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e"}
Sep 17 14:09:31.176855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:31.176840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerStarted","Data":"8235b46db1667e8e9c8c5f715f6771776de0383c6af57159d3e32f2edcea8d69"}
Sep 17 14:09:32.167095 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:32.167054    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:32.268527 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:32.268490    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:33.187093 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.187063    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd" event={"ID":"aa638175-c8b8-4ae6-8769-31d4ba1b0395","Type":"ContainerStarted","Data":"802ec255692ad84bb0652d054cec49d84c5892dd01323a9dd381c3604eab8b2e"}
Sep 17 14:09:33.187567 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.187139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd" event={"ID":"aa638175-c8b8-4ae6-8769-31d4ba1b0395","Type":"ContainerStarted","Data":"17268fe909320efe0bebfc8df864d8d9996ae93391d1af4ae65c8cd85fc35dd9"}
Sep 17 14:09:33.187567 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.187190    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:33.187567 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.187219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd" event={"ID":"aa638175-c8b8-4ae6-8769-31d4ba1b0395","Type":"ContainerStarted","Data":"690869cfa6d26ae3b9fa7f45554f7bc928428274a08ca3992f41409e1d0d3285"}
Sep 17 14:09:33.190375 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.190354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerStarted","Data":"2aaa9608402aebc111677a94722c0b699d96664f68d665af1e8c0630d0cd71ba"}
Sep 17 14:09:33.190463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.190404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerStarted","Data":"39e6aae60a5de94aa5fe397ead7790d1ccc49dbe3b00da962c030cf485403124"}
Sep 17 14:09:33.190463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.190429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerStarted","Data":"4ab59977464905b880b83cc8c64424a060220587e3cf66ebb4f45de597fada45"}
Sep 17 14:09:33.190463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.190454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerStarted","Data":"7f708233471308e8990f2f98089bfa283b919c1c33a3c5838cb6ad88ea06792d"}
Sep 17 14:09:33.190563 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.190477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerStarted","Data":"c4c37de2ff55c2d12f932c8746bc82ba34e1a5cc3fb016b09fb60cd159de66c9"}
Sep 17 14:09:33.190563 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.190499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerStarted","Data":"918e188f13a84d704a360f292a4b470739cf0a344e0275e63fca3345c13bc7eb"}
Sep 17 14:09:33.211208 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.211159    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd" podStartSLOduration=2.557687791 podStartE2EDuration="8.211141861s" totalImagesPullingTime="5.65345407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:33.210419125 +0000 UTC m=+160.143924316" watchObservedRunningTime="2026-09-17 14:09:33.211141861 +0000 UTC m=+160.144647051"
Sep 17 14:09:33.241955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.241911    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.762099807 podStartE2EDuration="8.241896894s" totalImagesPullingTime="6.459399158s" totalInitContainerRuntime="20.397929ms" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:25 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:33.240380785 +0000 UTC m=+160.173885977" watchObservedRunningTime="2026-09-17 14:09:33.241896894 +0000 UTC m=+160.175402091"
Sep 17 14:09:33.616988 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.616959    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c9ffcc65-7btds"]
Sep 17 14:09:33.651542 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.651515    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-f7757cb5f-qzn4b"]
Sep 17 14:09:33.654221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.654202    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.662481 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.662464    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 14:09:33.667757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.667738    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f7757cb5f-qzn4b"]
Sep 17 14:09:33.825670 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.825634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-service-ca\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.825670 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.825671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdnkm\" (UniqueName: \"kubernetes.io/projected/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-kube-api-access-tdnkm\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.825842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.825696    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-serving-cert\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.825842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.825823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-trusted-ca-bundle\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.825899 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.825854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-oauth-config\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.825899 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.825875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-config\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.825962 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.825908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-oauth-serving-cert\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.927314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.927231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-trusted-ca-bundle\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.927314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.927295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-oauth-config\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.927475 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.927456    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-config\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.927534 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.927489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-oauth-serving-cert\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.927534 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.927524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-service-ca\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.927623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.927563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdnkm\" (UniqueName: \"kubernetes.io/projected/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-kube-api-access-tdnkm\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.927623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.927600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-serving-cert\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.928180 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.928133    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-oauth-serving-cert\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.928290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.928202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-trusted-ca-bundle\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.928290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.928215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-config\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.928290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.928280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-service-ca\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.930023 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.930002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-oauth-config\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.930118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.930100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-serving-cert\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.935611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.935590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdnkm\" (UniqueName: \"kubernetes.io/projected/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-kube-api-access-tdnkm\") pod \"console-f7757cb5f-qzn4b\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") " pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:33.963657 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:33.963634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:34.028314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:34.028228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:09:34.030525 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:34.030499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3b860267-277a-4b1b-85b8-850e86d2cb3d-metrics-tls\") pod \"dns-default-flvcb\" (UID: \"3b860267-277a-4b1b-85b8-850e86d2cb3d\") " pod="openshift-dns/dns-default-flvcb"
Sep 17 14:09:34.200901 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:34.200854    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-85c49449dc-gqzzd"
Sep 17 14:09:34.233275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:34.233241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f7757cb5f-qzn4b"]
Sep 17 14:09:34.236391 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:09:34.236367    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b9f789_4c12_4b4c_8871_e9ecba5c10b9.slice/crio-4fb8875507ecb0bc2686c938a8398f7e6aed85ab7697b7fc9103b8f64dbd48b6 WatchSource:0}: Error finding container 4fb8875507ecb0bc2686c938a8398f7e6aed85ab7697b7fc9103b8f64dbd48b6: Status 404 returned error can't find the container with id 4fb8875507ecb0bc2686c938a8398f7e6aed85ab7697b7fc9103b8f64dbd48b6
Sep 17 14:09:34.269838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:34.269818    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jdh9k\""
Sep 17 14:09:34.277470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:34.277446    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-flvcb"
Sep 17 14:09:34.545687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:34.545663    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-flvcb"]
Sep 17 14:09:35.202608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:35.202568    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7757cb5f-qzn4b" event={"ID":"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9","Type":"ContainerStarted","Data":"08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b"}
Sep 17 14:09:35.203045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:35.202633    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7757cb5f-qzn4b" event={"ID":"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9","Type":"ContainerStarted","Data":"4fb8875507ecb0bc2686c938a8398f7e6aed85ab7697b7fc9103b8f64dbd48b6"}
Sep 17 14:09:35.204217 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:35.204191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flvcb" event={"ID":"3b860267-277a-4b1b-85b8-850e86d2cb3d","Type":"ContainerStarted","Data":"2764f724e9351dbd52c9bf4c670832f179dc2ac21b4ffd7ce647804694771d44"}
Sep 17 14:09:35.227174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:35.227121    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-f7757cb5f-qzn4b" podStartSLOduration=2.227104131 podStartE2EDuration="2.227104131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:35.224328357 +0000 UTC m=+162.157833547" watchObservedRunningTime="2026-09-17 14:09:35.227104131 +0000 UTC m=+162.160609322"
Sep 17 14:09:36.149747 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:36.149695    2565 patch_prober.go:28] interesting pod/image-registry-86df9b5656-bq6mt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 14:09:36.149919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:36.149781    2565 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-86df9b5656-bq6mt" podUID="51870341-f4ff-4804-98c3-31864864cc92" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:09:36.880936 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:36.880897    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:36.977829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:36.977798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:37.215796 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:37.215715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flvcb" event={"ID":"3b860267-277a-4b1b-85b8-850e86d2cb3d","Type":"ContainerStarted","Data":"67dc3c1d76dd38cf3edefcb979b5f89d2e55b14575062fcc03c1593bcb84546c"}
Sep 17 14:09:37.215796 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:37.215782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-flvcb" event={"ID":"3b860267-277a-4b1b-85b8-850e86d2cb3d","Type":"ContainerStarted","Data":"dde7985044e54e21550a2856c28bfbbde7790962bcb0a2ffd57c03654d1d8d7b"}
Sep 17 14:09:37.216010 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:37.215880    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-flvcb"
Sep 17 14:09:37.238893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:37.238839    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-flvcb" podStartSLOduration=130.380778054 podStartE2EDuration="2m12.238821044s" totalImagesPullingTime="1.85804299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:37.238156823 +0000 UTC m=+164.171662040" watchObservedRunningTime="2026-09-17 14:09:37.238821044 +0000 UTC m=+164.172326237"
Sep 17 14:09:38.110803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:38.110774    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-86df9b5656-bq6mt"
Sep 17 14:09:38.256536 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:38.256505    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:38.357725 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:38.357694    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:38.465994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:38.465961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:39.204674 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.204642    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:39.227485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.227458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerStarted","Data":"3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84"}
Sep 17 14:09:39.227599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.227544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerStarted","Data":"732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30"}
Sep 17 14:09:39.227599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.227575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerStarted","Data":"4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc"}
Sep 17 14:09:39.227673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.227599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerStarted","Data":"cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb"}
Sep 17 14:09:39.227673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.227622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerStarted","Data":"9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67"}
Sep 17 14:09:39.227673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.227645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerStarted","Data":"89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70"}
Sep 17 14:09:39.262115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.262073    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.190380877 podStartE2EDuration="9.262059948s" totalImagesPullingTime="7.049376614s" totalInitContainerRuntime="22.302457ms" isStatefulPod=false podCreationTimestamp="2026-09-17 14:09:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:09:39.260895442 +0000 UTC m=+166.194400633" watchObservedRunningTime="2026-09-17 14:09:39.262059948 +0000 UTC m=+166.195565169"
Sep 17 14:09:39.268243 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:39.268220    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:40.053079 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:40.053056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:40.153987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:40.153934    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:40.219027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:40.218987    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:40.615774 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:40.615753    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:09:43.964666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:43.964636    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:43.964666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:43.964680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:43.969000 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:43.968973    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:44.249656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:44.249601    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:09:44.407642 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:44.407601    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:44.506334 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:44.506213    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:44.578220 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:44.578181    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:45.452562 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:45.452531    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:45.531069 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:45.531036    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:47.222286 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:47.222194    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-flvcb"
Sep 17 14:09:51.861535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:51.861500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:51.964248 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:51.964210    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:54.589891 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:54.589856    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:54.687216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:54.687186    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:56.432109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:56.432077    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:56.534002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:56.533972    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:58.635668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.635597    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59c9ffcc65-7btds" podUID="48aefd83-e480-4b9e-bae8-92d9c88798bf" containerName="console" containerID="cri-o://8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8" gracePeriod=15
Sep 17 14:09:58.874758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.874731    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:58.884702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.884672    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:58.919702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.919650    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7bbn\" (UniqueName: \"kubernetes.io/projected/48aefd83-e480-4b9e-bae8-92d9c88798bf-kube-api-access-c7bbn\") pod \"48aefd83-e480-4b9e-bae8-92d9c88798bf\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") "
Sep 17 14:09:58.919702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.919686    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-config\") pod \"48aefd83-e480-4b9e-bae8-92d9c88798bf\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") "
Sep 17 14:09:58.919842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.919726    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-oauth-config\") pod \"48aefd83-e480-4b9e-bae8-92d9c88798bf\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") "
Sep 17 14:09:58.919842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.919758    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-serving-cert\") pod \"48aefd83-e480-4b9e-bae8-92d9c88798bf\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") "
Sep 17 14:09:58.919842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.919787    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-service-ca\") pod \"48aefd83-e480-4b9e-bae8-92d9c88798bf\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") "
Sep 17 14:09:58.919991 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.919975    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-oauth-serving-cert\") pod \"48aefd83-e480-4b9e-bae8-92d9c88798bf\" (UID: \"48aefd83-e480-4b9e-bae8-92d9c88798bf\") "
Sep 17 14:09:58.920223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.920127    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-config" (OuterVolumeSpecName: "console-config") pod "48aefd83-e480-4b9e-bae8-92d9c88798bf" (UID: "48aefd83-e480-4b9e-bae8-92d9c88798bf"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:09:58.920223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.920138    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-service-ca" (OuterVolumeSpecName: "service-ca") pod "48aefd83-e480-4b9e-bae8-92d9c88798bf" (UID: "48aefd83-e480-4b9e-bae8-92d9c88798bf"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:09:58.920403 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.920380    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-config\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:09:58.920461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.920407    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:09:58.920461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.920440    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "48aefd83-e480-4b9e-bae8-92d9c88798bf" (UID: "48aefd83-e480-4b9e-bae8-92d9c88798bf"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:09:58.922126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.922102    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48aefd83-e480-4b9e-bae8-92d9c88798bf-kube-api-access-c7bbn" (OuterVolumeSpecName: "kube-api-access-c7bbn") pod "48aefd83-e480-4b9e-bae8-92d9c88798bf" (UID: "48aefd83-e480-4b9e-bae8-92d9c88798bf"). InnerVolumeSpecName "kube-api-access-c7bbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:09:58.922203 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.922129    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "48aefd83-e480-4b9e-bae8-92d9c88798bf" (UID: "48aefd83-e480-4b9e-bae8-92d9c88798bf"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:09:58.922285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:58.922242    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "48aefd83-e480-4b9e-bae8-92d9c88798bf" (UID: "48aefd83-e480-4b9e-bae8-92d9c88798bf"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:09:59.019435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.019409    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:59.021589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.021568    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-oauth-config\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:09:59.021678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.021592    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/48aefd83-e480-4b9e-bae8-92d9c88798bf-console-serving-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:09:59.021678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.021605    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/48aefd83-e480-4b9e-bae8-92d9c88798bf-oauth-serving-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:09:59.021678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.021617    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7bbn\" (UniqueName: \"kubernetes.io/projected/48aefd83-e480-4b9e-bae8-92d9c88798bf-kube-api-access-c7bbn\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:09:59.306576 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.306504    2565 generic.go:356] "Generic (PLEG): container finished" podID="48aefd83-e480-4b9e-bae8-92d9c88798bf" containerID="8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8" exitCode=0
Sep 17 14:09:59.306576 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.306560    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c9ffcc65-7btds"
Sep 17 14:09:59.306576 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.306571    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c9ffcc65-7btds" event={"ID":"48aefd83-e480-4b9e-bae8-92d9c88798bf","Type":"ContainerDied","Data":"8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8"}
Sep 17 14:09:59.306735 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.306642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c9ffcc65-7btds" event={"ID":"48aefd83-e480-4b9e-bae8-92d9c88798bf","Type":"ContainerDied","Data":"01d64a803b4313622831b7c8cda37b1ac58e0fe73db8b416d27178060dcce9b6"}
Sep 17 14:09:59.306735 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.306665    2565 scope.go:117] "RemoveContainer" containerID="8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8"
Sep 17 14:09:59.316803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.316779    2565 scope.go:117] "RemoveContainer" containerID="8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8"
Sep 17 14:09:59.317152 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:09:59.317110    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8\": container with ID starting with 8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8 not found: ID does not exist" containerID="8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8"
Sep 17 14:09:59.317228 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.317159    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8"} err="failed to get container status \"8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8\": rpc error: code = NotFound desc = could not find container \"8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8\": container with ID starting with 8c609e629a614b08444a0bb0bbf361dcc920374565c6d7b92ff4b479e2a784b8 not found: ID does not exist"
Sep 17 14:09:59.321298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.321274    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:09:59.340374 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.340348    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59c9ffcc65-7btds"]
Sep 17 14:09:59.355580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.355555    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59c9ffcc65-7btds"]
Sep 17 14:09:59.518768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:09:59.518729    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48aefd83-e480-4b9e-bae8-92d9c88798bf" path="/var/lib/kubelet/pods/48aefd83-e480-4b9e-bae8-92d9c88798bf/volumes"
Sep 17 14:10:05.438626 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:05.438589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:05.539682 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:05.539650    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:06.848357 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:06.848323    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:06.948850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:06.948821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:09.352780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:09.352740    2565 generic.go:356] "Generic (PLEG): container finished" podID="ebf0ae63-0557-403a-a3ee-13969694249d" containerID="4e9fecd8cd53b32cbff6ac8f52884e7c6bc1d4eef6644731b0947ad75e99bcc3" exitCode=0
Sep 17 14:10:09.352780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:09.352781    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt" event={"ID":"ebf0ae63-0557-403a-a3ee-13969694249d","Type":"ContainerDied","Data":"4e9fecd8cd53b32cbff6ac8f52884e7c6bc1d4eef6644731b0947ad75e99bcc3"}
Sep 17 14:10:09.353182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:09.353150    2565 scope.go:117] "RemoveContainer" containerID="4e9fecd8cd53b32cbff6ac8f52884e7c6bc1d4eef6644731b0947ad75e99bcc3"
Sep 17 14:10:10.358733 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:10.358698    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-chdpt" event={"ID":"ebf0ae63-0557-403a-a3ee-13969694249d","Type":"ContainerStarted","Data":"c405cc41fe34a55e10c359290b07c29dda3d0965b50d3347a61c652774470a96"}
Sep 17 14:10:13.372965 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:13.372931    2565 generic.go:356] "Generic (PLEG): container finished" podID="50c4e844-c224-42f8-9d28-6f5c733038e8" containerID="6fcff2e07074f447b39cb2b091bf46955393f36c54e322e6ae6e04250d2c7660" exitCode=0
Sep 17 14:10:13.373302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:13.373004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fbr4k" event={"ID":"50c4e844-c224-42f8-9d28-6f5c733038e8","Type":"ContainerDied","Data":"6fcff2e07074f447b39cb2b091bf46955393f36c54e322e6ae6e04250d2c7660"}
Sep 17 14:10:13.373383 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:13.373369    2565 scope.go:117] "RemoveContainer" containerID="6fcff2e07074f447b39cb2b091bf46955393f36c54e322e6ae6e04250d2c7660"
Sep 17 14:10:14.378909 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:14.378871    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-fbr4k" event={"ID":"50c4e844-c224-42f8-9d28-6f5c733038e8","Type":"ContainerStarted","Data":"c7ceb41b28b89fb1f74e4f10686c7c7e3ebada0d85d0457af4c0f5247c750569"}
Sep 17 14:10:22.285214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:22.285175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:30.615883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:30.615810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:30.631561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:30.631524    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:31.470301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:31.470273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:43.313780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:43.313740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:43.762630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:43.762599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:44.363215 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.363181    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:10:44.363620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.363473    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="alertmanager" containerID="cri-o://918e188f13a84d704a360f292a4b470739cf0a344e0275e63fca3345c13bc7eb" gracePeriod=120
Sep 17 14:10:44.363620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.363545    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy-metric" containerID="cri-o://39e6aae60a5de94aa5fe397ead7790d1ccc49dbe3b00da962c030cf485403124" gracePeriod=120
Sep 17 14:10:44.363620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.363569    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy-web" containerID="cri-o://7f708233471308e8990f2f98089bfa283b919c1c33a3c5838cb6ad88ea06792d" gracePeriod=120
Sep 17 14:10:44.363797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.363598    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="config-reloader" containerID="cri-o://c4c37de2ff55c2d12f932c8746bc82ba34e1a5cc3fb016b09fb60cd159de66c9" gracePeriod=120
Sep 17 14:10:44.363797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.363625    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="prom-label-proxy" containerID="cri-o://2aaa9608402aebc111677a94722c0b699d96664f68d665af1e8c0630d0cd71ba" gracePeriod=120
Sep 17 14:10:44.363797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.363616    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy" containerID="cri-o://4ab59977464905b880b83cc8c64424a060220587e3cf66ebb4f45de597fada45" gracePeriod=120
Sep 17 14:10:44.514310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.514274    2565 generic.go:356] "Generic (PLEG): container finished" podID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerID="2aaa9608402aebc111677a94722c0b699d96664f68d665af1e8c0630d0cd71ba" exitCode=0
Sep 17 14:10:44.514310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.514297    2565 generic.go:356] "Generic (PLEG): container finished" podID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerID="4ab59977464905b880b83cc8c64424a060220587e3cf66ebb4f45de597fada45" exitCode=0
Sep 17 14:10:44.514310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.514307    2565 generic.go:356] "Generic (PLEG): container finished" podID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerID="c4c37de2ff55c2d12f932c8746bc82ba34e1a5cc3fb016b09fb60cd159de66c9" exitCode=0
Sep 17 14:10:44.514480 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.514317    2565 generic.go:356] "Generic (PLEG): container finished" podID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerID="918e188f13a84d704a360f292a4b470739cf0a344e0275e63fca3345c13bc7eb" exitCode=0
Sep 17 14:10:44.514480 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.514335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerDied","Data":"2aaa9608402aebc111677a94722c0b699d96664f68d665af1e8c0630d0cd71ba"}
Sep 17 14:10:44.514480 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.514406    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerDied","Data":"4ab59977464905b880b83cc8c64424a060220587e3cf66ebb4f45de597fada45"}
Sep 17 14:10:44.514480 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.514432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerDied","Data":"c4c37de2ff55c2d12f932c8746bc82ba34e1a5cc3fb016b09fb60cd159de66c9"}
Sep 17 14:10:44.514480 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:44.514457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerDied","Data":"918e188f13a84d704a360f292a4b470739cf0a344e0275e63fca3345c13bc7eb"}
Sep 17 14:10:45.522506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.522468    2565 generic.go:356] "Generic (PLEG): container finished" podID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerID="39e6aae60a5de94aa5fe397ead7790d1ccc49dbe3b00da962c030cf485403124" exitCode=0
Sep 17 14:10:45.522506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.522503    2565 generic.go:356] "Generic (PLEG): container finished" podID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerID="7f708233471308e8990f2f98089bfa283b919c1c33a3c5838cb6ad88ea06792d" exitCode=0
Sep 17 14:10:45.522870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.522533    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerDied","Data":"39e6aae60a5de94aa5fe397ead7790d1ccc49dbe3b00da962c030cf485403124"}
Sep 17 14:10:45.522870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.522596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerDied","Data":"7f708233471308e8990f2f98089bfa283b919c1c33a3c5838cb6ad88ea06792d"}
Sep 17 14:10:45.620348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.620324    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:10:45.791788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.791703    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-trusted-ca-bundle\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.791788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.791750    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-metrics-client-ca\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.791788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.791780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-out\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.791799    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-cluster-tls-config\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.791817    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.791840    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-volume\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.791864    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-web-config\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.791881    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nt9ld\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-kube-api-access-nt9ld\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792157    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:45.792423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792290    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:45.792423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-main-tls\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-tls-assets\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-web\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792497    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-main-db\") pod \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\" (UID: \"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6\") "
Sep 17 14:10:45.792814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792796    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.792869 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.792817    2565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-metrics-client-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.793172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.793144    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:10:45.795072 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.795041    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-kube-api-access-nt9ld" (OuterVolumeSpecName: "kube-api-access-nt9ld") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "kube-api-access-nt9ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:10:45.795314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.795288    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:45.795394 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.795284    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-out" (OuterVolumeSpecName: "config-out") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:10:45.795453 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.795405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:45.795673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.795643    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-volume" (OuterVolumeSpecName: "config-volume") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:45.795859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.795832    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:45.797247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.797229    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:10:45.797373 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.797357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:45.799323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.799242    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:45.806379 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.806355    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-web-config" (OuterVolumeSpecName: "web-config") pod "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" (UID: "c10f32b4-c9b2-406f-b88c-a45e9e43e8a6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:45.893670 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893643    2565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-cluster-tls-config\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893669    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893686    2565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893720    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-web-config\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893734    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt9ld\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-kube-api-access-nt9ld\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893749    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893764    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-main-tls\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893777    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-tls-assets\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893790    2565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893803    2565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-alertmanager-main-db\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:45.893983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:45.893817    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6-config-out\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:46.529545 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.529514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c10f32b4-c9b2-406f-b88c-a45e9e43e8a6","Type":"ContainerDied","Data":"91dcded040b14c52548eaa662f87675c677b0d4567cf7110fc9cc8421274de09"}
Sep 17 14:10:46.529849 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.529562    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 17 14:10:46.529849 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.529587    2565 scope.go:117] "RemoveContainer" containerID="2aaa9608402aebc111677a94722c0b699d96664f68d665af1e8c0630d0cd71ba"
Sep 17 14:10:46.537051 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.537028    2565 scope.go:117] "RemoveContainer" containerID="39e6aae60a5de94aa5fe397ead7790d1ccc49dbe3b00da962c030cf485403124"
Sep 17 14:10:46.546659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.546641    2565 scope.go:117] "RemoveContainer" containerID="4ab59977464905b880b83cc8c64424a060220587e3cf66ebb4f45de597fada45"
Sep 17 14:10:46.553395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.553371    2565 scope.go:117] "RemoveContainer" containerID="7f708233471308e8990f2f98089bfa283b919c1c33a3c5838cb6ad88ea06792d"
Sep 17 14:10:46.555894 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.555871    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:10:46.559762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.559740    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 17 14:10:46.561524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.561506    2565 scope.go:117] "RemoveContainer" containerID="c4c37de2ff55c2d12f932c8746bc82ba34e1a5cc3fb016b09fb60cd159de66c9"
Sep 17 14:10:46.568179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.568162    2565 scope.go:117] "RemoveContainer" containerID="918e188f13a84d704a360f292a4b470739cf0a344e0275e63fca3345c13bc7eb"
Sep 17 14:10:46.574889 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:46.574874    2565 scope.go:117] "RemoveContainer" containerID="44ae9a325b3f1fd876a61423f8e11a9ffcd0bf8ee1bdd12cfdf76c9cbde19b8f"
Sep 17 14:10:47.518992 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:47.518957    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" path="/var/lib/kubelet/pods/c10f32b4-c9b2-406f-b88c-a45e9e43e8a6/volumes"
Sep 17 14:10:48.768600 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:48.768557    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:10:48.768994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:48.768883    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="prometheus" containerID="cri-o://89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70" gracePeriod=600
Sep 17 14:10:48.768994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:48.768930    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84" gracePeriod=600
Sep 17 14:10:48.768994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:48.768937    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="thanos-sidecar" containerID="cri-o://cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb" gracePeriod=600
Sep 17 14:10:48.768994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:48.768964    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy-web" containerID="cri-o://4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc" gracePeriod=600
Sep 17 14:10:48.768994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:48.768898    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy" containerID="cri-o://732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30" gracePeriod=600
Sep 17 14:10:48.769303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:48.768952    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="config-reloader" containerID="cri-o://9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67" gracePeriod=600
Sep 17 14:10:49.545036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545003    2565 generic.go:356] "Generic (PLEG): container finished" podID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerID="3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84" exitCode=0
Sep 17 14:10:49.545036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545028    2565 generic.go:356] "Generic (PLEG): container finished" podID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerID="732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30" exitCode=0
Sep 17 14:10:49.545036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545034    2565 generic.go:356] "Generic (PLEG): container finished" podID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerID="cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb" exitCode=0
Sep 17 14:10:49.545036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545041    2565 generic.go:356] "Generic (PLEG): container finished" podID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerID="9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67" exitCode=0
Sep 17 14:10:49.545036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545046    2565 generic.go:356] "Generic (PLEG): container finished" podID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerID="89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70" exitCode=0
Sep 17 14:10:49.545421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545078    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerDied","Data":"3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84"}
Sep 17 14:10:49.545421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerDied","Data":"732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30"}
Sep 17 14:10:49.545421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerDied","Data":"cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb"}
Sep 17 14:10:49.545421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerDied","Data":"9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67"}
Sep 17 14:10:49.545421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:49.545300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerDied","Data":"89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70"}
Sep 17 14:10:50.018470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.018446    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.026394 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026373    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-web-config\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026407    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-serving-certs-ca-bundle\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-tls\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026447    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-tls-assets\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026473    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-kube-rbac-proxy\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026500    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-config\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026531    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-0\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026594    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-db\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026641    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-metrics-client-ca\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026706    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-2\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026743    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-metrics-client-certs\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026765    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:50.026785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026771    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-kubelet-serving-ca-bundle\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.027210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026837    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-1\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.027210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026863    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-grpc-tls\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.027210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026896    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-config-out\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.027210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-trusted-ca-bundle\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.027210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.026970    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-thanos-prometheus-http-client-file\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.027210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.027011    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bh5wb\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-kube-api-access-bh5wb\") pod \"66999a84-0b96-49ca-8a59-6ffc91f76963\" (UID: \"66999a84-0b96-49ca-8a59-6ffc91f76963\") "
Sep 17 14:10:50.027210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.027139    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:50.027669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.027300    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.027669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.027447    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:50.028180 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.027947    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:50.029239 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.029141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:50.029239 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.029202    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:50.029687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.029551    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:10:50.029687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.029657    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:10:50.030016 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.029989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.030299 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.030235    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:10:50.030637 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.030582    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.030872 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.030835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.031106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.031057    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-config" (OuterVolumeSpecName: "config") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.031196 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.031163    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.031736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.031709    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.031847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.031715    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-kube-api-access-bh5wb" (OuterVolumeSpecName: "kube-api-access-bh5wb") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "kube-api-access-bh5wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:10:50.032109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.032077    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.032247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.032220    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-config-out" (OuterVolumeSpecName: "config-out") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:10:50.032637 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.032610    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.040156 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.040131    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-web-config" (OuterVolumeSpecName: "web-config") pod "66999a84-0b96-49ca-8a59-6ffc91f76963" (UID: "66999a84-0b96-49ca-8a59-6ffc91f76963"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:10:50.128466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128393    2565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-metrics-client-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128418    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128428    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128437    2565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-grpc-tls\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128447    2565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-config-out\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128455    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128464    2565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128472    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh5wb\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-kube-api-access-bh5wb\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128481    2565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-web-config\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128489    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128497    2565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66999a84-0b96-49ca-8a59-6ffc91f76963-tls-assets\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128505    2565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-kube-rbac-proxy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128514    2565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-config\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128522    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128530    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128538    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-db\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128548    2565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66999a84-0b96-49ca-8a59-6ffc91f76963-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128557    2565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-configmap-metrics-client-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.128776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.128567    2565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/66999a84-0b96-49ca-8a59-6ffc91f76963-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:10:50.292364 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.292331    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:50.351118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.351083    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:50.556397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.556326    2565 generic.go:356] "Generic (PLEG): container finished" podID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerID="4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc" exitCode=0
Sep 17 14:10:50.556516 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.556409    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.556516 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.556408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerDied","Data":"4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc"}
Sep 17 14:10:50.556602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.556588    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"66999a84-0b96-49ca-8a59-6ffc91f76963","Type":"ContainerDied","Data":"8235b46db1667e8e9c8c5f715f6771776de0383c6af57159d3e32f2edcea8d69"}
Sep 17 14:10:50.556654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.556640    2565 scope.go:117] "RemoveContainer" containerID="3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84"
Sep 17 14:10:50.565024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.564884    2565 scope.go:117] "RemoveContainer" containerID="732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30"
Sep 17 14:10:50.571844 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.571827    2565 scope.go:117] "RemoveContainer" containerID="4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc"
Sep 17 14:10:50.578284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.578267    2565 scope.go:117] "RemoveContainer" containerID="cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb"
Sep 17 14:10:50.581030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.581007    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:10:50.585139 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.585114    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:10:50.586058 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.586042    2565 scope.go:117] "RemoveContainer" containerID="9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67"
Sep 17 14:10:50.592474 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.592457    2565 scope.go:117] "RemoveContainer" containerID="89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70"
Sep 17 14:10:50.599379 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.599361    2565 scope.go:117] "RemoveContainer" containerID="8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e"
Sep 17 14:10:50.605809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.605793    2565 scope.go:117] "RemoveContainer" containerID="3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84"
Sep 17 14:10:50.606188 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:10:50.606166    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84\": container with ID starting with 3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84 not found: ID does not exist" containerID="3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84"
Sep 17 14:10:50.606279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.606195    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84"} err="failed to get container status \"3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84\": rpc error: code = NotFound desc = could not find container \"3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84\": container with ID starting with 3691685045e0a5c917b7e4c5d6a9ce0dc100f9c20a7f0c43a489254a230dee84 not found: ID does not exist"
Sep 17 14:10:50.606279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.606212    2565 scope.go:117] "RemoveContainer" containerID="732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30"
Sep 17 14:10:50.606495 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:10:50.606478    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30\": container with ID starting with 732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30 not found: ID does not exist" containerID="732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30"
Sep 17 14:10:50.606553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.606499    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30"} err="failed to get container status \"732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30\": rpc error: code = NotFound desc = could not find container \"732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30\": container with ID starting with 732b10bd09fc0a53a34486242087ecbd9e77b88c03a80435f07d163d1a64ea30 not found: ID does not exist"
Sep 17 14:10:50.606553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.606513    2565 scope.go:117] "RemoveContainer" containerID="4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc"
Sep 17 14:10:50.606759 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:10:50.606745    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc\": container with ID starting with 4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc not found: ID does not exist" containerID="4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc"
Sep 17 14:10:50.606797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.606761    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc"} err="failed to get container status \"4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc\": rpc error: code = NotFound desc = could not find container \"4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc\": container with ID starting with 4a3c704845c5fe01962d3430a299a0942bfa2611aa993e0d95068882edddfafc not found: ID does not exist"
Sep 17 14:10:50.606797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.606773    2565 scope.go:117] "RemoveContainer" containerID="cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb"
Sep 17 14:10:50.606988 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:10:50.606970    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb\": container with ID starting with cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb not found: ID does not exist" containerID="cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb"
Sep 17 14:10:50.607028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.606994    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb"} err="failed to get container status \"cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb\": rpc error: code = NotFound desc = could not find container \"cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb\": container with ID starting with cfc089db5e6cf6ae000f6dc569bcd205a1eb81801c1e663dd5c3ff6d203aa1bb not found: ID does not exist"
Sep 17 14:10:50.607028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.607011    2565 scope.go:117] "RemoveContainer" containerID="9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67"
Sep 17 14:10:50.607245 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:10:50.607227    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67\": container with ID starting with 9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67 not found: ID does not exist" containerID="9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67"
Sep 17 14:10:50.607347 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.607323    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67"} err="failed to get container status \"9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67\": rpc error: code = NotFound desc = could not find container \"9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67\": container with ID starting with 9072e370bf3b8ac4e3dc1ecf55c60f34c7a54d2e726f73752d2ed4371e480b67 not found: ID does not exist"
Sep 17 14:10:50.607410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.607351    2565 scope.go:117] "RemoveContainer" containerID="89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70"
Sep 17 14:10:50.607604 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:10:50.607588    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70\": container with ID starting with 89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70 not found: ID does not exist" containerID="89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70"
Sep 17 14:10:50.607646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.607608    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70"} err="failed to get container status \"89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70\": rpc error: code = NotFound desc = could not find container \"89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70\": container with ID starting with 89850f5b06084833dea79f26b7a0c6024bcbd11cb99dad581397a948745e7b70 not found: ID does not exist"
Sep 17 14:10:50.607646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.607621    2565 scope.go:117] "RemoveContainer" containerID="8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e"
Sep 17 14:10:50.607834 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:10:50.607816    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e\": container with ID starting with 8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e not found: ID does not exist" containerID="8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e"
Sep 17 14:10:50.607886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.607839    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e"} err="failed to get container status \"8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e\": rpc error: code = NotFound desc = could not find container \"8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e\": container with ID starting with 8f7956639dea7e6d803a40df7e7dbc870e8f1e5cc72b7440be547d80c950922e not found: ID does not exist"
Sep 17 14:10:50.616223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616200    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:10:50.616530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616513    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy"
Sep 17 14:10:50.616610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616531    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy"
Sep 17 14:10:50.616610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616546    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="init-config-reloader"
Sep 17 14:10:50.616610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616554    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="init-config-reloader"
Sep 17 14:10:50.616610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616569    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="alertmanager"
Sep 17 14:10:50.616610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616586    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="alertmanager"
Sep 17 14:10:50.616610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616596    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="thanos-sidecar"
Sep 17 14:10:50.616610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616606    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="thanos-sidecar"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616617    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="init-config-reloader"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616626    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="init-config-reloader"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616636    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy-web"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616644    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy-web"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616653    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616661    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616670    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48aefd83-e480-4b9e-bae8-92d9c88798bf" containerName="console"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616678    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48aefd83-e480-4b9e-bae8-92d9c88798bf" containerName="console"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616691    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy-metric"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616699    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy-metric"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616708    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="prom-label-proxy"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616717    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="prom-label-proxy"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616732    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="config-reloader"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616740    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="config-reloader"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616748    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy-web"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616756    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy-web"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616767    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="config-reloader"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616775    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="config-reloader"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616784    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy-thanos"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616793    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy-thanos"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616807    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="prometheus"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616815    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="prometheus"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616894    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="alertmanager"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616906    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="prometheus"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616915    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616925    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy-web"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616935    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="config-reloader"
Sep 17 14:10:50.616931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616944    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy-web"
Sep 17 14:10:50.618210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616952    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy-thanos"
Sep 17 14:10:50.618210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616960    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="kube-rbac-proxy-metric"
Sep 17 14:10:50.618210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616968    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c10f32b4-c9b2-406f-b88c-a45e9e43e8a6" containerName="prom-label-proxy"
Sep 17 14:10:50.618210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616977    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="config-reloader"
Sep 17 14:10:50.618210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616985    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="kube-rbac-proxy"
Sep 17 14:10:50.618210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.616995    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="48aefd83-e480-4b9e-bae8-92d9c88798bf" containerName="console"
Sep 17 14:10:50.618210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.617002    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" containerName="thanos-sidecar"
Sep 17 14:10:50.621159 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.621142    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.623580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.623561    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 17 14:10:50.623667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.623563    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-v9bsz\""
Sep 17 14:10:50.623667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.623612    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 17 14:10:50.623773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.623683    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 17 14:10:50.623922 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.623908    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-9jefi79h9ugaa\""
Sep 17 14:10:50.623962 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.623938    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 17 14:10:50.624166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.624149    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 17 14:10:50.624306 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.624288    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 17 14:10:50.624890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.624872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 14:10:50.624973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.624948    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 17 14:10:50.625291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.625276    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 17 14:10:50.625516 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.625498    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 17 14:10:50.625583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.625517    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 17 14:10:50.625583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.625500    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 17 14:10:50.625784 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.625761    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 17 14:10:50.626366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.626350    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 17 14:10:50.630599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.630579    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 17 14:10:50.632265 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632321    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632425 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-config\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632425 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632428    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632456    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1f677c8-e587-4279-a153-bf9180c8fece-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632495    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632609    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1f677c8-e587-4279-a153-bf9180c8fece-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632684    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4j6v\" (UniqueName: \"kubernetes.io/projected/f1f677c8-e587-4279-a153-bf9180c8fece-kube-api-access-d4j6v\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.632850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.632755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.636633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.636611    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:10:50.733231 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1f677c8-e587-4279-a153-bf9180c8fece-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733355 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733355 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733355 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1f677c8-e587-4279-a153-bf9180c8fece-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4j6v\" (UniqueName: \"kubernetes.io/projected/f1f677c8-e587-4279-a153-bf9180c8fece-kube-api-access-d4j6v\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733561    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733674    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-config\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.733807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733796    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.734303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.733907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.734303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.734092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.734725 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.734608    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.736100 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.735280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.736100 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.735560    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.737973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.736460    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.737973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.736567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.737973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.736956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.737973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.737399    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-web-config\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.737973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.737561    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.737973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.737587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1f677c8-e587-4279-a153-bf9180c8fece-config-out\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.737973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.737913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1f677c8-e587-4279-a153-bf9180c8fece-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.738378 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.738032    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.738378 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.738231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.738378 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.738312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.738378 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.738366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.738682 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.738658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.739188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.739169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f1f677c8-e587-4279-a153-bf9180c8fece-config\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.739278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.739171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f1f677c8-e587-4279-a153-bf9180c8fece-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.742848 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.742826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4j6v\" (UniqueName: \"kubernetes.io/projected/f1f677c8-e587-4279-a153-bf9180c8fece-kube-api-access-d4j6v\") pod \"prometheus-k8s-0\" (UID: \"f1f677c8-e587-4279-a153-bf9180c8fece\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:50.930334 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:50.930308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:10:51.215547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:51.215519    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 17 14:10:51.217237 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:10:51.217205    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f677c8_e587_4279_a153_bf9180c8fece.slice/crio-c48adec50e13c324475fd3f56e2840f984b4fced9ce7b25823277c005039b308 WatchSource:0}: Error finding container c48adec50e13c324475fd3f56e2840f984b4fced9ce7b25823277c005039b308: Status 404 returned error can't find the container with id c48adec50e13c324475fd3f56e2840f984b4fced9ce7b25823277c005039b308
Sep 17 14:10:51.518801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:51.518736    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66999a84-0b96-49ca-8a59-6ffc91f76963" path="/var/lib/kubelet/pods/66999a84-0b96-49ca-8a59-6ffc91f76963/volumes"
Sep 17 14:10:51.562550 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:51.562524    2565 generic.go:356] "Generic (PLEG): container finished" podID="f1f677c8-e587-4279-a153-bf9180c8fece" containerID="aacfe61e6671ef50d828c0b17b8a864bba53a8b52e02290730e861c7f97135f0" exitCode=0
Sep 17 14:10:51.562699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:51.562576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1f677c8-e587-4279-a153-bf9180c8fece","Type":"ContainerDied","Data":"aacfe61e6671ef50d828c0b17b8a864bba53a8b52e02290730e861c7f97135f0"}
Sep 17 14:10:51.562699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:51.562626    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1f677c8-e587-4279-a153-bf9180c8fece","Type":"ContainerStarted","Data":"c48adec50e13c324475fd3f56e2840f984b4fced9ce7b25823277c005039b308"}
Sep 17 14:10:52.362402 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.362365    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:52.509280 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.509235    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:10:52.570895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.570845    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1f677c8-e587-4279-a153-bf9180c8fece","Type":"ContainerStarted","Data":"379dc49f0bcd73539cfc28e267993367b6bee462bd902d329c48d037e3618d5c"}
Sep 17 14:10:52.571032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.570966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1f677c8-e587-4279-a153-bf9180c8fece","Type":"ContainerStarted","Data":"ac895529d15297b4bf723d6f4da37961981a7b7422e1c63a7d5df4c7f76cc362"}
Sep 17 14:10:52.571032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.570995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1f677c8-e587-4279-a153-bf9180c8fece","Type":"ContainerStarted","Data":"8ddc87f17917ebace33d0477c109a1178232995ac4208bab5dbf1c6439f37160"}
Sep 17 14:10:52.571123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.571035    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1f677c8-e587-4279-a153-bf9180c8fece","Type":"ContainerStarted","Data":"364f97677b3f1e7f79b1ff691ad786d78edafa32d2ba6185a8e95a656cb8fb48"}
Sep 17 14:10:52.571123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.571079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1f677c8-e587-4279-a153-bf9180c8fece","Type":"ContainerStarted","Data":"b4f8bf5844e25862702176dddedc2d873ab310d845f84b762d60ec392beeabae"}
Sep 17 14:10:52.571123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.571107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f1f677c8-e587-4279-a153-bf9180c8fece","Type":"ContainerStarted","Data":"86b4b232d33dabb3110795b9e1a23c1539f826c1148ee5364e85a581b7ac0a28"}
Sep 17 14:10:52.627042 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:52.626944    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.605417004 podStartE2EDuration="2.626929247s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.512243ms" isStatefulPod=false podCreationTimestamp="2026-09-17 14:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:10:52.6256603 +0000 UTC m=+239.559165530" watchObservedRunningTime="2026-09-17 14:10:52.626929247 +0000 UTC m=+239.560434473"
Sep 17 14:10:55.930717 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:10:55.930684    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:11:03.238726 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:03.238688    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f7757cb5f-qzn4b"]
Sep 17 14:11:03.359087 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:03.359054    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:03.549716 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:03.549641    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:03.653814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:03.653771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:28.259282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.259161    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-f7757cb5f-qzn4b" podUID="f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" containerName="console" containerID="cri-o://08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b" gracePeriod=15
Sep 17 14:11:28.503094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.503064    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:11:28.633426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633400    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-serving-cert\") pod \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") "
Sep 17 14:11:28.633543 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633445    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-oauth-config\") pod \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") "
Sep 17 14:11:28.633543 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633480    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdnkm\" (UniqueName: \"kubernetes.io/projected/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-kube-api-access-tdnkm\") pod \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") "
Sep 17 14:11:28.633543 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633499    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-service-ca\") pod \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") "
Sep 17 14:11:28.633543 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633524    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-trusted-ca-bundle\") pod \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") "
Sep 17 14:11:28.633543 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-config\") pod \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") "
Sep 17 14:11:28.633790 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633563    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-oauth-serving-cert\") pod \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\" (UID: \"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9\") "
Sep 17 14:11:28.633994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633947    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-config" (OuterVolumeSpecName: "console-config") pod "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" (UID: "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:11:28.633994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.633976    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-service-ca" (OuterVolumeSpecName: "service-ca") pod "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" (UID: "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:11:28.634137 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.634038    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" (UID: "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:11:28.634137 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.634060    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" (UID: "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:11:28.635630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.635602    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-kube-api-access-tdnkm" (OuterVolumeSpecName: "kube-api-access-tdnkm") pod "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" (UID: "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9"). InnerVolumeSpecName "kube-api-access-tdnkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:11:28.635938 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.635921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" (UID: "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:11:28.635990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.635951    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" (UID: "f9b9f789-4c12-4b4c-8871-e9ecba5c10b9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:11:28.706770 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.706744    2565 generic.go:356] "Generic (PLEG): container finished" podID="f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" containerID="08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b" exitCode=0
Sep 17 14:11:28.706842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.706788    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f7757cb5f-qzn4b"
Sep 17 14:11:28.706842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.706820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7757cb5f-qzn4b" event={"ID":"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9","Type":"ContainerDied","Data":"08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b"}
Sep 17 14:11:28.706922 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.706861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f7757cb5f-qzn4b" event={"ID":"f9b9f789-4c12-4b4c-8871-e9ecba5c10b9","Type":"ContainerDied","Data":"4fb8875507ecb0bc2686c938a8398f7e6aed85ab7697b7fc9103b8f64dbd48b6"}
Sep 17 14:11:28.706922 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.706882    2565 scope.go:117] "RemoveContainer" containerID="08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b"
Sep 17 14:11:28.714747 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.714728    2565 scope.go:117] "RemoveContainer" containerID="08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b"
Sep 17 14:11:28.715017 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:11:28.715000    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b\": container with ID starting with 08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b not found: ID does not exist" containerID="08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b"
Sep 17 14:11:28.715111 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.715023    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b"} err="failed to get container status \"08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b\": rpc error: code = NotFound desc = could not find container \"08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b\": container with ID starting with 08ea84dbd9be5acbbe347ce6316ba8cd5a74894f92f67673f45677c972dbba2b not found: ID does not exist"
Sep 17 14:11:28.731239 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.731216    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f7757cb5f-qzn4b"]
Sep 17 14:11:28.734273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.734228    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-serving-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:11:28.734273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.734266    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-oauth-config\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:11:28.734447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.734282    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdnkm\" (UniqueName: \"kubernetes.io/projected/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-kube-api-access-tdnkm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:11:28.734447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.734295    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:11:28.734447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.734307    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-trusted-ca-bundle\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:11:28.734447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.734319    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-console-config\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:11:28.734447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.734331    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9-oauth-serving-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:11:28.736264 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:28.736234    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f7757cb5f-qzn4b"]
Sep 17 14:11:29.518421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:29.518391    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" path="/var/lib/kubelet/pods/f9b9f789-4c12-4b4c-8871-e9ecba5c10b9/volumes"
Sep 17 14:11:29.584036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:29.584009    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:44.172946 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:44.172906    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:50.931220 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:50.931183    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:11:50.946399 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:50.946373    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:11:51.812034 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:51.812006    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 17 14:11:53.412356 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:53.412329    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:11:53.413146 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:53.413122    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:11:53.424925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:53.424887    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:11:53.425802 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:53.425784    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:11:53.430887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:53.430869    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 14:11:58.614985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:58.614944    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:11:58.679782 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:11:58.679749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:07.234396 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:07.234360    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:07.330323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:07.330295    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:08.309471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:08.309434    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:08.416266 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:08.416219    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:10.630006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:10.629969    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:10.697225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:10.697191    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:17.638951 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:17.638916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:17.736499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:17.736454    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:17.799196 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:17.799169    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:21.858858 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:21.858827    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:21.954861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:21.954831    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:45.150813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:45.150771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:12:45.252497 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:12:45.252464    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:15.410514 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:13:15.410472    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:22.211797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:13:22.211760    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:22.752298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:13:22.752245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:54.224227 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:13:54.224187    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:56.425464 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:13:56.425405    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:57.841023 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:13:57.840986    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:13:57.945921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:13:57.945890    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:00.206770 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:00.206731    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:00.313515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:00.313483    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:00.459115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:00.459039    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:02.423588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:02.423555    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:02.625445 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:02.625416    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:04.237986 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:04.237946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:04.352496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:04.352462    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:10.871236 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:10.871206    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:12.517628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:12.517592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:12.664712 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:12.664682    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:13.967939 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:13.967903    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"]
Sep 17 14:14:13.968345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:13.968204    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" containerName="console"
Sep 17 14:14:13.968345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:13.968216    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" containerName="console"
Sep 17 14:14:13.968345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:13.968290    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9b9f789-4c12-4b4c-8871-e9ecba5c10b9" containerName="console"
Sep 17 14:14:13.970032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:13.970014    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"
Sep 17 14:14:13.976485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:13.976073    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-r4vmn\""
Sep 17 14:14:13.976485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:13.976093    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 14:14:13.976485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:13.976221    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 14:14:14.004773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.004748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"]
Sep 17 14:14:14.073863 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.073836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d2f6h\" (UID: \"1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"
Sep 17 14:14:14.073974 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.073876    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwf5\" (UniqueName: \"kubernetes.io/projected/1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f-kube-api-access-xdwf5\") pod \"openshift-lws-operator-fd8ccff4c-d2f6h\" (UID: \"1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"
Sep 17 14:14:14.174595 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.174567    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d2f6h\" (UID: \"1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"
Sep 17 14:14:14.174682 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.174610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwf5\" (UniqueName: \"kubernetes.io/projected/1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f-kube-api-access-xdwf5\") pod \"openshift-lws-operator-fd8ccff4c-d2f6h\" (UID: \"1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"
Sep 17 14:14:14.174977 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.174959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-d2f6h\" (UID: \"1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"
Sep 17 14:14:14.183022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.183004    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwf5\" (UniqueName: \"kubernetes.io/projected/1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f-kube-api-access-xdwf5\") pod \"openshift-lws-operator-fd8ccff4c-d2f6h\" (UID: \"1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"
Sep 17 14:14:14.287543 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.287494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"
Sep 17 14:14:14.558814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.558725    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h"]
Sep 17 14:14:14.561416 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:14:14.561389    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b2a2bee_4d16_4a8a_b2d0_16b0b498eb4f.slice/crio-ca23c00c5cbbac45acaa2344dbfccf21df37179ed325ff17c909026d39158426 WatchSource:0}: Error finding container ca23c00c5cbbac45acaa2344dbfccf21df37179ed325ff17c909026d39158426: Status 404 returned error can't find the container with id ca23c00c5cbbac45acaa2344dbfccf21df37179ed325ff17c909026d39158426
Sep 17 14:14:14.562824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:14.562806    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:14:15.367028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:15.366988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h" event={"ID":"1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f","Type":"ContainerStarted","Data":"ca23c00c5cbbac45acaa2344dbfccf21df37179ed325ff17c909026d39158426"}
Sep 17 14:14:17.378407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:17.378365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h" event={"ID":"1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f","Type":"ContainerStarted","Data":"01a468a4d0e852ceb900bae489e70cad2fb1866ce5a89dfaa15419e2df101faf"}
Sep 17 14:14:17.398209 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:17.398145    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-d2f6h" podStartSLOduration=1.7926642369999999 podStartE2EDuration="4.398127334s" totalImagesPullingTime="2.605463097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:17.39632026 +0000 UTC m=+444.329825454" watchObservedRunningTime="2026-09-17 14:14:17.398127334 +0000 UTC m=+444.331632528"
Sep 17 14:14:18.105221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:18.103681    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:18.208123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:18.208092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:20.215003 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:20.214970    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:28.211860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:28.211829    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:28.279407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:28.279375    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:36.053177 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:36.053143    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:36.144699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:36.144652    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:36.830314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:36.830281    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:37.052445 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.052410    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:37.149287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.149199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:37.293501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.293474    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"]
Sep 17 14:14:37.296834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.296818    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.299146 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.299126    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fx4tc\""
Sep 17 14:14:37.299266 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.299237    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 14:14:37.302540 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.302521    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 14:14:37.302669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.302566    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 14:14:37.302669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.302622    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 14:14:37.338453 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.338425    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"]
Sep 17 14:14:37.347829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.347806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59172329-6d9b-48e6-85f1-6ba9226f6cee-webhook-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.347941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.347854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59172329-6d9b-48e6-85f1-6ba9226f6cee-tmp\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.347941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.347871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59172329-6d9b-48e6-85f1-6ba9226f6cee-manifests\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.348063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.347968    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59172329-6d9b-48e6-85f1-6ba9226f6cee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.348063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.348010    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6fqw\" (UniqueName: \"kubernetes.io/projected/59172329-6d9b-48e6-85f1-6ba9226f6cee-kube-api-access-t6fqw\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.446095 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.446032    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:37.449133 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.449115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59172329-6d9b-48e6-85f1-6ba9226f6cee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.449192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.449145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6fqw\" (UniqueName: \"kubernetes.io/projected/59172329-6d9b-48e6-85f1-6ba9226f6cee-kube-api-access-t6fqw\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.449192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.449178    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59172329-6d9b-48e6-85f1-6ba9226f6cee-webhook-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.449290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.449245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59172329-6d9b-48e6-85f1-6ba9226f6cee-tmp\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.449349 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.449294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59172329-6d9b-48e6-85f1-6ba9226f6cee-manifests\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.449642 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.449617    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/59172329-6d9b-48e6-85f1-6ba9226f6cee-manifests\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.451457 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.451436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/59172329-6d9b-48e6-85f1-6ba9226f6cee-tmp\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.451709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.451587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/59172329-6d9b-48e6-85f1-6ba9226f6cee-webhook-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.451780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.451753    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/59172329-6d9b-48e6-85f1-6ba9226f6cee-apiservice-cert\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.458339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.458319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6fqw\" (UniqueName: \"kubernetes.io/projected/59172329-6d9b-48e6-85f1-6ba9226f6cee-kube-api-access-t6fqw\") pod \"opendatahub-operator-controller-manager-569db487f9-x9mlw\" (UID: \"59172329-6d9b-48e6-85f1-6ba9226f6cee\") " pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.553305 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.553283    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:37.607432 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.607411    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:37.675923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.675880    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:37.772675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.772604    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:37.874441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.874412    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:37.894696 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.894671    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"]
Sep 17 14:14:37.897808 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:14:37.897781    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59172329_6d9b_48e6_85f1_6ba9226f6cee.slice/crio-1e5fbd28c9d369f82312f36b091b36160190de2f7c0a2b103394f03d5ab322b3 WatchSource:0}: Error finding container 1e5fbd28c9d369f82312f36b091b36160190de2f7c0a2b103394f03d5ab322b3: Status 404 returned error can't find the container with id 1e5fbd28c9d369f82312f36b091b36160190de2f7c0a2b103394f03d5ab322b3
Sep 17 14:14:37.949822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:37.949801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:38.462125 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:38.462051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw" event={"ID":"59172329-6d9b-48e6-85f1-6ba9226f6cee","Type":"ContainerStarted","Data":"1e5fbd28c9d369f82312f36b091b36160190de2f7c0a2b103394f03d5ab322b3"}
Sep 17 14:14:41.477801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:41.477767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw" event={"ID":"59172329-6d9b-48e6-85f1-6ba9226f6cee","Type":"ContainerStarted","Data":"58bf876bbc5631dd8776581d196637c6324a9f8e008814d5aed2c8bbbbf719e0"}
Sep 17 14:14:42.483816 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:42.483782    2565 generic.go:356] "Generic (PLEG): container finished" podID="59172329-6d9b-48e6-85f1-6ba9226f6cee" containerID="58bf876bbc5631dd8776581d196637c6324a9f8e008814d5aed2c8bbbbf719e0" exitCode=0
Sep 17 14:14:42.484172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:42.483836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw" event={"ID":"59172329-6d9b-48e6-85f1-6ba9226f6cee","Type":"ContainerDied","Data":"58bf876bbc5631dd8776581d196637c6324a9f8e008814d5aed2c8bbbbf719e0"}
Sep 17 14:14:43.490438 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:43.490398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw" event={"ID":"59172329-6d9b-48e6-85f1-6ba9226f6cee","Type":"ContainerStarted","Data":"3b567249300b7f46b079bf16d3c3a940e998255a4bf64e41ab7045afb4dedea8"}
Sep 17 14:14:43.490810 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:43.490490    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:43.514820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:43.514774    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw" podStartSLOduration=2.095951794 podStartE2EDuration="6.514759288s" totalImagesPullingTime="3.381191362s" totalInitContainerRuntime="1.037616132s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:43.514344692 +0000 UTC m=+470.447849884" watchObservedRunningTime="2026-09-17 14:14:43.514759288 +0000 UTC m=+470.448264480"
Sep 17 14:14:48.981160 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:48.981122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:49.073694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:49.073662    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:51.682442 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:51.682410    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:51.842456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:51.842424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:54.497862 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:54.497832    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-569db487f9-x9mlw"
Sep 17 14:14:55.604785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.604756    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"]
Sep 17 14:14:55.611988 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.611966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.617040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.617011    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-gbr9s\""
Sep 17 14:14:55.617040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.617017    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 14:14:55.617192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.617058    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 14:14:55.617192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.617157    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 14:14:55.628952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.628932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"]
Sep 17 14:14:55.697440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.697411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4631013b-cd4a-46c2-b1d1-e53523819969-manager-config\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.697440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.697442    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4631013b-cd4a-46c2-b1d1-e53523819969-cert\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.697611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.697477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4631013b-cd4a-46c2-b1d1-e53523819969-metrics-cert\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.697611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.697503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2prb\" (UniqueName: \"kubernetes.io/projected/4631013b-cd4a-46c2-b1d1-e53523819969-kube-api-access-v2prb\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.798394 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.798367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4631013b-cd4a-46c2-b1d1-e53523819969-manager-config\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.798513 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.798401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4631013b-cd4a-46c2-b1d1-e53523819969-cert\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.798513 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.798473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4631013b-cd4a-46c2-b1d1-e53523819969-metrics-cert\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.798513 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.798496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2prb\" (UniqueName: \"kubernetes.io/projected/4631013b-cd4a-46c2-b1d1-e53523819969-kube-api-access-v2prb\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.799099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.799075    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4631013b-cd4a-46c2-b1d1-e53523819969-manager-config\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.800825 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.800803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4631013b-cd4a-46c2-b1d1-e53523819969-cert\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.800910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.800872    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4631013b-cd4a-46c2-b1d1-e53523819969-metrics-cert\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.817999 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.817968    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2prb\" (UniqueName: \"kubernetes.io/projected/4631013b-cd4a-46c2-b1d1-e53523819969-kube-api-access-v2prb\") pod \"lws-controller-manager-77fdd77d78-n87hl\" (UID: \"4631013b-cd4a-46c2-b1d1-e53523819969\") " pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:55.921614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:55.921560    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:56.224077 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:56.224052    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"]
Sep 17 14:14:56.226246 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:14:56.226222    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4631013b_cd4a_46c2_b1d1_e53523819969.slice/crio-d9fea69e70979d3b137f9bc6ca92325e8f9d89be2adba59a52acb4c5c4561d5a WatchSource:0}: Error finding container d9fea69e70979d3b137f9bc6ca92325e8f9d89be2adba59a52acb4c5c4561d5a: Status 404 returned error can't find the container with id d9fea69e70979d3b137f9bc6ca92325e8f9d89be2adba59a52acb4c5c4561d5a
Sep 17 14:14:56.546467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:56.546397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl" event={"ID":"4631013b-cd4a-46c2-b1d1-e53523819969","Type":"ContainerStarted","Data":"d9fea69e70979d3b137f9bc6ca92325e8f9d89be2adba59a52acb4c5c4561d5a"}
Sep 17 14:14:56.926933 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:56.926898    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:57.061485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:57.061449    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:57.371234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:57.371196    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:57.887393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:57.887358    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:58.556968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:58.556930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl" event={"ID":"4631013b-cd4a-46c2-b1d1-e53523819969","Type":"ContainerStarted","Data":"816a40eeafc22764f437834c3c594e180682e3ed6fa779fa6b0664aabbf38d39"}
Sep 17 14:14:58.557435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:58.557062    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:14:58.580727 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:58.580680    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl" podStartSLOduration=2.278117586 podStartE2EDuration="3.580660172s" totalImagesPullingTime="1.302542586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:14:58.579174456 +0000 UTC m=+485.512679649" watchObservedRunningTime="2026-09-17 14:14:58.580660172 +0000 UTC m=+485.514165364"
Sep 17 14:14:59.076552 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:59.076517    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:14:59.192883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:14:59.192848    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:00.879706 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:00.879671    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:03.935498 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:03.935462    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:04.034940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.034910    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:04.772794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.772762    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"]
Sep 17 14:15:04.776075 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.776050    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.778177 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.778149    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-47xhr\""
Sep 17 14:15:04.778342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.778153    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 14:15:04.788922 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.788902    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"]
Sep 17 14:15:04.823131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.823089    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:04.870121 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de956cc-9510-458d-84db-b1883388a194-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.870220 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.870220 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.870315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.870315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.870380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de956cc-9510-458d-84db-b1883388a194-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.870380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de956cc-9510-458d-84db-b1883388a194-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.870380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mcx\" (UniqueName: \"kubernetes.io/projected/9de956cc-9510-458d-84db-b1883388a194-kube-api-access-p7mcx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.870474 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.870410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.971593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de956cc-9510-458d-84db-b1883388a194-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971599    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971652    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de956cc-9510-458d-84db-b1883388a194-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de956cc-9510-458d-84db-b1883388a194-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mcx\" (UniqueName: \"kubernetes.io/projected/9de956cc-9510-458d-84db-b1883388a194-kube-api-access-p7mcx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.971778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.972056    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.972197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.972227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.972312    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.972628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.972609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de956cc-9510-458d-84db-b1883388a194-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.974214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.974191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de956cc-9510-458d-84db-b1883388a194-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.974342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.974268    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de956cc-9510-458d-84db-b1883388a194-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.985410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.985385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mcx\" (UniqueName: \"kubernetes.io/projected/9de956cc-9510-458d-84db-b1883388a194-kube-api-access-p7mcx\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:04.989294 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:04.989272    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de956cc-9510-458d-84db-b1883388a194-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2\" (UID: \"9de956cc-9510-458d-84db-b1883388a194\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:05.038170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:05.038124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:05.085979 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:05.085959    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:05.363760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:05.363589    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"]
Sep 17 14:15:05.366247 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:15:05.366214    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de956cc_9510_458d_84db_b1883388a194.slice/crio-4ce298a9e38797195d87ff08e74347b96f88eef0a9588489d541429e654ce594 WatchSource:0}: Error finding container 4ce298a9e38797195d87ff08e74347b96f88eef0a9588489d541429e654ce594: Status 404 returned error can't find the container with id 4ce298a9e38797195d87ff08e74347b96f88eef0a9588489d541429e654ce594
Sep 17 14:15:05.587123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:05.587079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2" event={"ID":"9de956cc-9510-458d-84db-b1883388a194","Type":"ContainerStarted","Data":"4ce298a9e38797195d87ff08e74347b96f88eef0a9588489d541429e654ce594"}
Sep 17 14:15:05.868385 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:05.868350    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:07.753771 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:07.753730    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:07.754032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:07.753811    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:07.754032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:07.753841    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:15:08.020488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:08.020385    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:08.115319 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:08.115290    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:08.216592 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:08.216559    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:08.561601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:08.561567    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:08.604022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:08.603990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2" event={"ID":"9de956cc-9510-458d-84db-b1883388a194","Type":"ContainerStarted","Data":"6d77920dadda5bc8039e2f03cb0fb0574f95f396d7519d3a9188be12c3d73792"}
Sep 17 14:15:08.630754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:08.630709    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2" podStartSLOduration=2.245273233 podStartE2EDuration="4.630693559s" totalImagesPullingTime="2.385420326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:08.628277906 +0000 UTC m=+495.561783099" watchObservedRunningTime="2026-09-17 14:15:08.630693559 +0000 UTC m=+495.564198751"
Sep 17 14:15:08.678688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:08.678660    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:09.086897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:09.086860    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:09.091504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:09.091482    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:09.176222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:09.176193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:09.564024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:09.563931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77fdd77d78-n87hl"
Sep 17 14:15:09.608270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:09.608224    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:09.609191 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:09.609167    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2"
Sep 17 14:15:19.101660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.101616    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:19.214646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.214615    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"]
Sep 17 14:15:19.217921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.217901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:19.221643 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.221623    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lvqql\""
Sep 17 14:15:19.223078 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.223055    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 17 14:15:19.225610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.225590    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:19.233938 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.233913    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"]
Sep 17 14:15:19.398360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.398275    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34172f79-3f60-4ccf-919a-deb390f30b02-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kndh5\" (UID: \"34172f79-3f60-4ccf-919a-deb390f30b02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:19.398360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.398344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6llf\" (UniqueName: \"kubernetes.io/projected/34172f79-3f60-4ccf-919a-deb390f30b02-kube-api-access-b6llf\") pod \"llmisvc-controller-manager-777c6bd7d5-kndh5\" (UID: \"34172f79-3f60-4ccf-919a-deb390f30b02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:19.499606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.499578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6llf\" (UniqueName: \"kubernetes.io/projected/34172f79-3f60-4ccf-919a-deb390f30b02-kube-api-access-b6llf\") pod \"llmisvc-controller-manager-777c6bd7d5-kndh5\" (UID: \"34172f79-3f60-4ccf-919a-deb390f30b02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:19.499711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.499638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34172f79-3f60-4ccf-919a-deb390f30b02-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kndh5\" (UID: \"34172f79-3f60-4ccf-919a-deb390f30b02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:19.499750 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:15:19.499725    2565 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 17 14:15:19.499801 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:15:19.499790    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/34172f79-3f60-4ccf-919a-deb390f30b02-cert podName:34172f79-3f60-4ccf-919a-deb390f30b02 nodeName:}" failed. No retries permitted until 2026-09-17 14:15:19.999773782 +0000 UTC m=+506.933278952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/34172f79-3f60-4ccf-919a-deb390f30b02-cert") pod "llmisvc-controller-manager-777c6bd7d5-kndh5" (UID: "34172f79-3f60-4ccf-919a-deb390f30b02") : secret "llmisvc-webhook-server-cert" not found
Sep 17 14:15:19.530776 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.530749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6llf\" (UniqueName: \"kubernetes.io/projected/34172f79-3f60-4ccf-919a-deb390f30b02-kube-api-access-b6llf\") pod \"llmisvc-controller-manager-777c6bd7d5-kndh5\" (UID: \"34172f79-3f60-4ccf-919a-deb390f30b02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:19.593271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.593229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:19.715773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:19.715720    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:15:20.003274 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:20.003183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34172f79-3f60-4ccf-919a-deb390f30b02-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kndh5\" (UID: \"34172f79-3f60-4ccf-919a-deb390f30b02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:20.005478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:20.005455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/34172f79-3f60-4ccf-919a-deb390f30b02-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-kndh5\" (UID: \"34172f79-3f60-4ccf-919a-deb390f30b02\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:20.128955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:20.128914    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:20.399296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:20.399269    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"]
Sep 17 14:15:20.401694 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:15:20.401665    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod34172f79_3f60_4ccf_919a_deb390f30b02.slice/crio-4d94d99f3613b056da28cf48f70d632f496baf7cd368e3cbc1493e217c0f6c4e WatchSource:0}: Error finding container 4d94d99f3613b056da28cf48f70d632f496baf7cd368e3cbc1493e217c0f6c4e: Status 404 returned error can't find the container with id 4d94d99f3613b056da28cf48f70d632f496baf7cd368e3cbc1493e217c0f6c4e
Sep 17 14:15:20.657285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:20.657200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5" event={"ID":"34172f79-3f60-4ccf-919a-deb390f30b02","Type":"ContainerStarted","Data":"4d94d99f3613b056da28cf48f70d632f496baf7cd368e3cbc1493e217c0f6c4e"}
Sep 17 14:15:23.679609 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:23.679578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5" event={"ID":"34172f79-3f60-4ccf-919a-deb390f30b02","Type":"ContainerStarted","Data":"0da7302134b1ad8eb2d300034d741807882e374505a207f48cf67bdf53cd0283"}
Sep 17 14:15:23.679969 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:23.679671    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:15:23.702355 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:23.702274    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5" podStartSLOduration=1.6629741999999998 podStartE2EDuration="4.702247667s" totalImagesPullingTime="3.039273467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:15:23.70193788 +0000 UTC m=+510.635443072" watchObservedRunningTime="2026-09-17 14:15:23.702247667 +0000 UTC m=+510.635752862"
Sep 17 14:15:54.685759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:15:54.685686    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-kndh5"
Sep 17 14:16:09.003979 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:09.003941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:09.100314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:09.100283    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:23.285656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:23.285622    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:23.923377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:23.923345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:24.741212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:24.741171    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:25.546924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:25.546892    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:27.789722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:27.789686    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:27.935238 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:27.935207    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:28.569693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.569654    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"]
Sep 17 14:16:28.572040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.572017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:28.580050 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:28.580021    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-xk98g\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xk98g\"" type="*v1.Secret"
Sep 17 14:16:28.580156 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:28.580059    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:28.580156 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:28.580075    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 14:16:28.580316 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:28.580234    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 14:16:28.609896 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.609873    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"]
Sep 17 14:16:28.730201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.730164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rdxh\" (UniqueName: \"kubernetes.io/projected/7c495b28-bd62-4a56-b431-e816b4264856-kube-api-access-7rdxh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" (UID: \"7c495b28-bd62-4a56-b431-e816b4264856\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:28.730396 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.730221    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c495b28-bd62-4a56-b431-e816b4264856-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" (UID: \"7c495b28-bd62-4a56-b431-e816b4264856\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:28.746567 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.746543    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:28.831681 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.831646    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c495b28-bd62-4a56-b431-e816b4264856-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" (UID: \"7c495b28-bd62-4a56-b431-e816b4264856\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:28.832110 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.831773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rdxh\" (UniqueName: \"kubernetes.io/projected/7c495b28-bd62-4a56-b431-e816b4264856-kube-api-access-7rdxh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" (UID: \"7c495b28-bd62-4a56-b431-e816b4264856\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:28.832110 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:28.832009    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c495b28-bd62-4a56-b431-e816b4264856-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" (UID: \"7c495b28-bd62-4a56-b431-e816b4264856\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:29.085171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:29.085092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:29.338942 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:29.338853    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:29.675004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:29.674918    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 14:16:29.872745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:29.872715    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xk98g\""
Sep 17 14:16:29.924371 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:29.924347    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 14:16:29.936452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:29.936389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rdxh\" (UniqueName: \"kubernetes.io/projected/7c495b28-bd62-4a56-b431-e816b4264856-kube-api-access-7rdxh\") pod \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" (UID: \"7c495b28-bd62-4a56-b431-e816b4264856\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:30.082438 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:30.082409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:30.364810 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:30.361083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"]
Sep 17 14:16:30.370548 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:16:30.370513    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c495b28_bd62_4a56_b431_e816b4264856.slice/crio-5be3bb7fc9634ebb2bc91833ad703631e79a866d53ebd78beaa8da495e660e70 WatchSource:0}: Error finding container 5be3bb7fc9634ebb2bc91833ad703631e79a866d53ebd78beaa8da495e660e70: Status 404 returned error can't find the container with id 5be3bb7fc9634ebb2bc91833ad703631e79a866d53ebd78beaa8da495e660e70
Sep 17 14:16:30.977284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:30.977237    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d" event={"ID":"7c495b28-bd62-4a56-b431-e816b4264856","Type":"ContainerStarted","Data":"5be3bb7fc9634ebb2bc91833ad703631e79a866d53ebd78beaa8da495e660e70"}
Sep 17 14:16:31.785119 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:31.785073    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:33.610699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:33.610620    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:33.759119 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:33.759070    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:35.006074 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:35.006036    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:37.008270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:37.008226    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d" event={"ID":"7c495b28-bd62-4a56-b431-e816b4264856","Type":"ContainerStarted","Data":"b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c"}
Sep 17 14:16:37.008660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:37.008365    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:37.073271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:37.070784    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d" podStartSLOduration=3.311533935 podStartE2EDuration="9.070749829s" totalImagesPullingTime="5.759215894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:37.066699319 +0000 UTC m=+584.000204508" watchObservedRunningTime="2026-09-17 14:16:37.070749829 +0000 UTC m=+584.004255023"
Sep 17 14:16:38.906457 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:38.906422    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:39.017807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.017776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:39.176585 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.176505    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"]
Sep 17 14:16:39.178975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.178954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.182432 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.182406    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-g5n8f\""
Sep 17 14:16:39.182603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.182580    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 14:16:39.182672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.182594    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 14:16:39.198568 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.198542    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"]
Sep 17 14:16:39.206575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.206554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:39.220123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.220094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt9dq\" (UniqueName: \"kubernetes.io/projected/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-kube-api-access-vt9dq\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.220224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.220146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.220224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.220164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.301119 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.301095    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:39.321371 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.321343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt9dq\" (UniqueName: \"kubernetes.io/projected/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-kube-api-access-vt9dq\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.321614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.321389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.321614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.321408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.322021 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.321999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.323621 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.323593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.333950 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.333929    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt9dq\" (UniqueName: \"kubernetes.io/projected/9c6e6391-e1b9-4ae5-ad32-406cda3a137f-kube-api-access-vt9dq\") pod \"kuadrant-console-plugin-fbdd4f445-wr52n\" (UID: \"9c6e6391-e1b9-4ae5-ad32-406cda3a137f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.406227 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.406201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:39.487481 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.487427    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"
Sep 17 14:16:39.716165 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.716123    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:39.773640 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.773614    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n"]
Sep 17 14:16:39.773876 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:16:39.773853    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c6e6391_e1b9_4ae5_ad32_406cda3a137f.slice/crio-b0e50b136e93d6c17de27afb60e5e01ef291e6da3d09f45285e58cfff38a0c87 WatchSource:0}: Error finding container b0e50b136e93d6c17de27afb60e5e01ef291e6da3d09f45285e58cfff38a0c87: Status 404 returned error can't find the container with id b0e50b136e93d6c17de27afb60e5e01ef291e6da3d09f45285e58cfff38a0c87
Sep 17 14:16:39.871227 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:39.871202    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:40.022590 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:40.022510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n" event={"ID":"9c6e6391-e1b9-4ae5-ad32-406cda3a137f","Type":"ContainerStarted","Data":"b0e50b136e93d6c17de27afb60e5e01ef291e6da3d09f45285e58cfff38a0c87"}
Sep 17 14:16:45.050990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:45.050950    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n" event={"ID":"9c6e6391-e1b9-4ae5-ad32-406cda3a137f","Type":"ContainerStarted","Data":"5e2c0d05bf2730eac3f727cb43068c22fd05e4c3285ba746019c574751038231"}
Sep 17 14:16:45.083419 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:45.083364    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-wr52n" podStartSLOduration=1.777409419 podStartE2EDuration="6.083344482s" totalImagesPullingTime="4.305935063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:45.082506112 +0000 UTC m=+592.016011304" watchObservedRunningTime="2026-09-17 14:16:45.083344482 +0000 UTC m=+592.016849682"
Sep 17 14:16:47.910152 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:47.910109    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:48.015286 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:48.015235    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:48.197328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:48.197205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:50.486875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:50.486839    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:50.690638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:50.690605    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:51.621924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.621894    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"]
Sep 17 14:16:51.622377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.622051    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d" podUID="7c495b28-bd62-4a56-b431-e816b4264856" containerName="manager" containerID="cri-o://b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c" gracePeriod=2
Sep 17 14:16:51.633461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.633432    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"]
Sep 17 14:16:51.645323 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:51.645295    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:51.757961 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.757930    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"]
Sep 17 14:16:51.758461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.758440    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c495b28-bd62-4a56-b431-e816b4264856" containerName="manager"
Sep 17 14:16:51.758461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.758462    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c495b28-bd62-4a56-b431-e816b4264856" containerName="manager"
Sep 17 14:16:51.758651 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.758566    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c495b28-bd62-4a56-b431-e816b4264856" containerName="manager"
Sep 17 14:16:51.761728 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.761703    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:51.783192 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:51.783162    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:51.789148 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.789122    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"]
Sep 17 14:16:51.827726 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.827701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs956\" (UniqueName: \"kubernetes.io/projected/b9141668-faee-4053-9523-8e9df4b2c6e9-kube-api-access-rs956\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5p57\" (UID: \"b9141668-faee-4053-9523-8e9df4b2c6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:51.827856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.827763    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9141668-faee-4053-9523-8e9df4b2c6e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5p57\" (UID: \"b9141668-faee-4053-9523-8e9df4b2c6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:51.873808 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.873756    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"]
Sep 17 14:16:51.877014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.876987    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:51.879790 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.879769    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:51.885308 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:51.885277    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-lnxfz\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:51.898526 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:51.898493    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:51.920366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.920343    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"]
Sep 17 14:16:51.928493 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.928467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rdxh\" (UniqueName: \"kubernetes.io/projected/7c495b28-bd62-4a56-b431-e816b4264856-kube-api-access-7rdxh\") pod \"7c495b28-bd62-4a56-b431-e816b4264856\" (UID: \"7c495b28-bd62-4a56-b431-e816b4264856\") "
Sep 17 14:16:51.928599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.928504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c495b28-bd62-4a56-b431-e816b4264856-extensions-socket-volume\") pod \"7c495b28-bd62-4a56-b431-e816b4264856\" (UID: \"7c495b28-bd62-4a56-b431-e816b4264856\") "
Sep 17 14:16:51.928730 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.928713    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs956\" (UniqueName: \"kubernetes.io/projected/b9141668-faee-4053-9523-8e9df4b2c6e9-kube-api-access-rs956\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5p57\" (UID: \"b9141668-faee-4053-9523-8e9df4b2c6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:51.928798 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.928766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9141668-faee-4053-9523-8e9df4b2c6e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5p57\" (UID: \"b9141668-faee-4053-9523-8e9df4b2c6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:51.928855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.928823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thqwr\" (UniqueName: \"kubernetes.io/projected/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-kube-api-access-thqwr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnxfz\" (UID: \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:51.928911 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.928880    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnxfz\" (UID: \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:51.929084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.929059    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c495b28-bd62-4a56-b431-e816b4264856-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "7c495b28-bd62-4a56-b431-e816b4264856" (UID: "7c495b28-bd62-4a56-b431-e816b4264856"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:16:51.929201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.929178    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9141668-faee-4053-9523-8e9df4b2c6e9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5p57\" (UID: \"b9141668-faee-4053-9523-8e9df4b2c6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:51.930733 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.930705    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c495b28-bd62-4a56-b431-e816b4264856-kube-api-access-7rdxh" (OuterVolumeSpecName: "kube-api-access-7rdxh") pod "7c495b28-bd62-4a56-b431-e816b4264856" (UID: "7c495b28-bd62-4a56-b431-e816b4264856"). InnerVolumeSpecName "kube-api-access-7rdxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:16:51.971344 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:51.971312    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:51.988645 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:51.988623    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs956\" (UniqueName: \"kubernetes.io/projected/b9141668-faee-4053-9523-8e9df4b2c6e9-kube-api-access-rs956\") pod \"kuadrant-operator-controller-manager-76f8f686bf-f5p57\" (UID: \"b9141668-faee-4053-9523-8e9df4b2c6e9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:52.030086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.030064    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thqwr\" (UniqueName: \"kubernetes.io/projected/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-kube-api-access-thqwr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnxfz\" (UID: \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:52.030172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.030098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnxfz\" (UID: \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:52.030172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.030139    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rdxh\" (UniqueName: \"kubernetes.io/projected/7c495b28-bd62-4a56-b431-e816b4264856-kube-api-access-7rdxh\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:16:52.030172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.030148    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/7c495b28-bd62-4a56-b431-e816b4264856-extensions-socket-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:16:52.030426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.030411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnxfz\" (UID: \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:52.055953 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.055934    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thqwr\" (UniqueName: \"kubernetes.io/projected/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-kube-api-access-thqwr\") pod \"kuadrant-operator-controller-manager-5769f99d9f-lnxfz\" (UID: \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:52.075926 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.075905    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:52.082523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.082499    2565 generic.go:356] "Generic (PLEG): container finished" podID="7c495b28-bd62-4a56-b431-e816b4264856" containerID="b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c" exitCode=0
Sep 17 14:16:52.082623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.082552    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:52.082623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.082566    2565 scope.go:117] "RemoveContainer" containerID="b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c"
Sep 17 14:16:52.091728 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:52.091701    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:52.092370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.092352    2565 scope.go:117] "RemoveContainer" containerID="b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c"
Sep 17 14:16:52.092634 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:52.092617    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c\": container with ID starting with b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c not found: ID does not exist" containerID="b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c"
Sep 17 14:16:52.092705 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.092642    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c"} err="failed to get container status \"b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c\": rpc error: code = NotFound desc = could not find container \"b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c\": container with ID starting with b9950c585eca61f5e22a6375f126a16e1269f1898f07a7ef7b3e485fa362df7c not found: ID does not exist"
Sep 17 14:16:52.095717 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:52.095691    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:52.187829 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.187794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:52.379309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.379286    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"]
Sep 17 14:16:52.381868 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:16:52.381841    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9141668_faee_4053_9523_8e9df4b2c6e9.slice/crio-8b7629b08cdf58fccb86bfb275a6ff9ae68f8c02717e52920c3affba57fddf3e WatchSource:0}: Error finding container 8b7629b08cdf58fccb86bfb275a6ff9ae68f8c02717e52920c3affba57fddf3e: Status 404 returned error can't find the container with id 8b7629b08cdf58fccb86bfb275a6ff9ae68f8c02717e52920c3affba57fddf3e
Sep 17 14:16:52.473157 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:52.473130    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"]
Sep 17 14:16:52.475913 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:16:52.475889    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4317e5_a5c3_4e14_b889_0b45c3e4e082.slice/crio-b1e37834ac2e930ceb9cb3cebadccff285371290dfc97aca1dba5878e7075af4 WatchSource:0}: Error finding container b1e37834ac2e930ceb9cb3cebadccff285371290dfc97aca1dba5878e7075af4: Status 404 returned error can't find the container with id b1e37834ac2e930ceb9cb3cebadccff285371290dfc97aca1dba5878e7075af4
Sep 17 14:16:53.090025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.089988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57" event={"ID":"b9141668-faee-4053-9523-8e9df4b2c6e9","Type":"ContainerStarted","Data":"ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3"}
Sep 17 14:16:53.090501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.090174    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:16:53.090501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.090211    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57" event={"ID":"b9141668-faee-4053-9523-8e9df4b2c6e9","Type":"ContainerStarted","Data":"8b7629b08cdf58fccb86bfb275a6ff9ae68f8c02717e52920c3affba57fddf3e"}
Sep 17 14:16:53.091768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.091736    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz" event={"ID":"0c4317e5-a5c3-4e14-b889-0b45c3e4e082","Type":"ContainerStarted","Data":"4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b"}
Sep 17 14:16:53.091883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.091774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz" event={"ID":"0c4317e5-a5c3-4e14-b889-0b45c3e4e082","Type":"ContainerStarted","Data":"b1e37834ac2e930ceb9cb3cebadccff285371290dfc97aca1dba5878e7075af4"}
Sep 17 14:16:53.091883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.091804    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:16:53.093946 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:53.093919    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:53.152056 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:53.152019    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:53.152967 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.152889    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57" podStartSLOduration=2.152872019 podStartE2EDuration="2.152872019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:53.149582915 +0000 UTC m=+600.083088108" watchObservedRunningTime="2026-09-17 14:16:53.152872019 +0000 UTC m=+600.086377212"
Sep 17 14:16:53.182140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.182098    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz" podStartSLOduration=2.182076261 podStartE2EDuration="2.182076261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:16:53.180641157 +0000 UTC m=+600.114146349" watchObservedRunningTime="2026-09-17 14:16:53.182076261 +0000 UTC m=+600.115581452"
Sep 17 14:16:53.453349 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.453238    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:16:53.453894 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.453863    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:16:53.460507 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.460481    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:16:53.461146 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.461126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:16:53.519234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:53.519204    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c495b28-bd62-4a56-b431-e816b4264856" path="/var/lib/kubelet/pods/7c495b28-bd62-4a56-b431-e816b4264856/volumes"
Sep 17 14:16:53.521498 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:16:53.521467    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-mws7d\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="7c495b28-bd62-4a56-b431-e816b4264856" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-mws7d"
Sep 17 14:16:55.115273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:55.115224    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:16:55.270106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:16:55.270075    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:00.466391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:00.466355    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:00.592110 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:00.592084    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:00.731753 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:00.731686    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:00.894035 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:00.894004    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:04.097820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.097789    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:17:04.098188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.097857    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:17:04.304614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.304581    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"]
Sep 17 14:17:04.304803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.304769    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57" podUID="b9141668-faee-4053-9523-8e9df4b2c6e9" containerName="manager" containerID="cri-o://ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3" gracePeriod=10
Sep 17 14:17:04.548467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.548443    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:17:04.634094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.634068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rs956\" (UniqueName: \"kubernetes.io/projected/b9141668-faee-4053-9523-8e9df4b2c6e9-kube-api-access-rs956\") pod \"b9141668-faee-4053-9523-8e9df4b2c6e9\" (UID: \"b9141668-faee-4053-9523-8e9df4b2c6e9\") "
Sep 17 14:17:04.634309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.634292    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9141668-faee-4053-9523-8e9df4b2c6e9-extensions-socket-volume\") pod \"b9141668-faee-4053-9523-8e9df4b2c6e9\" (UID: \"b9141668-faee-4053-9523-8e9df4b2c6e9\") "
Sep 17 14:17:04.634636 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.634611    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9141668-faee-4053-9523-8e9df4b2c6e9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b9141668-faee-4053-9523-8e9df4b2c6e9" (UID: "b9141668-faee-4053-9523-8e9df4b2c6e9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:17:04.636553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.636524    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9141668-faee-4053-9523-8e9df4b2c6e9-kube-api-access-rs956" (OuterVolumeSpecName: "kube-api-access-rs956") pod "b9141668-faee-4053-9523-8e9df4b2c6e9" (UID: "b9141668-faee-4053-9523-8e9df4b2c6e9"). InnerVolumeSpecName "kube-api-access-rs956". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:04.735764 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.735689    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9141668-faee-4053-9523-8e9df4b2c6e9-extensions-socket-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:17:04.735764 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:04.735717    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs956\" (UniqueName: \"kubernetes.io/projected/b9141668-faee-4053-9523-8e9df4b2c6e9-kube-api-access-rs956\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:17:05.150157 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.150120    2565 generic.go:356] "Generic (PLEG): container finished" podID="b9141668-faee-4053-9523-8e9df4b2c6e9" containerID="ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3" exitCode=0
Sep 17 14:17:05.150575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.150169    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"
Sep 17 14:17:05.150575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.150178    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57" event={"ID":"b9141668-faee-4053-9523-8e9df4b2c6e9","Type":"ContainerDied","Data":"ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3"}
Sep 17 14:17:05.150575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.150217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57" event={"ID":"b9141668-faee-4053-9523-8e9df4b2c6e9","Type":"ContainerDied","Data":"8b7629b08cdf58fccb86bfb275a6ff9ae68f8c02717e52920c3affba57fddf3e"}
Sep 17 14:17:05.150575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.150239    2565 scope.go:117] "RemoveContainer" containerID="ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3"
Sep 17 14:17:05.158883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.158864    2565 scope.go:117] "RemoveContainer" containerID="ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3"
Sep 17 14:17:05.159177 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:17:05.159160    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3\": container with ID starting with ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3 not found: ID does not exist" containerID="ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3"
Sep 17 14:17:05.159217 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.159187    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3"} err="failed to get container status \"ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3\": rpc error: code = NotFound desc = could not find container \"ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3\": container with ID starting with ea65549649a9883a5407421a71a3a7e61412d1b90c1b4ac3217c70681f6561e3 not found: ID does not exist"
Sep 17 14:17:05.199686 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.199652    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"]
Sep 17 14:17:05.210761 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.210737    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-f5p57"]
Sep 17 14:17:05.518998 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:05.518907    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9141668-faee-4053-9523-8e9df4b2c6e9" path="/var/lib/kubelet/pods/b9141668-faee-4053-9523-8e9df4b2c6e9/volumes"
Sep 17 14:17:07.187556 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:07.187432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:07.361822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:07.361788    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:09.249985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:09.249944    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:09.358282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:09.358214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:11.685939 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:11.685905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:11.770627 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:11.770592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:16.365762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:16.365674    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:16.465231 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:16.465187    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:20.851981 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.851943    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"]
Sep 17 14:17:20.852397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.852331    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9141668-faee-4053-9523-8e9df4b2c6e9" containerName="manager"
Sep 17 14:17:20.852397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.852345    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9141668-faee-4053-9523-8e9df4b2c6e9" containerName="manager"
Sep 17 14:17:20.852477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.852439    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9141668-faee-4053-9523-8e9df4b2c6e9" containerName="manager"
Sep 17 14:17:20.854518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.854500    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.857679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.857657    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-75x8t\""
Sep 17 14:17:20.880238 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.880201    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"]
Sep 17 14:17:20.901511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.901473    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:20.977207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.977430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93debedb-47b6-4fd8-a144-c4c629341e42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.977430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.977430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njszn\" (UniqueName: \"kubernetes.io/projected/93debedb-47b6-4fd8-a144-c4c629341e42-kube-api-access-njszn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.977430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.977430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.977430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977427    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93debedb-47b6-4fd8-a144-c4c629341e42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.977667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:20.977667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:20.977481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93debedb-47b6-4fd8-a144-c4c629341e42-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.068850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.068796    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:21.078398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93debedb-47b6-4fd8-a144-c4c629341e42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078448    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93debedb-47b6-4fd8-a144-c4c629341e42-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93debedb-47b6-4fd8-a144-c4c629341e42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njszn\" (UniqueName: \"kubernetes.io/projected/93debedb-47b6-4fd8-a144-c4c629341e42-kube-api-access-njszn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.078982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.078912    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.079086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.079063    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.079270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.079163    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.079362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.079177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.079362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.079191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93debedb-47b6-4fd8-a144-c4c629341e42-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.080804 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.080784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93debedb-47b6-4fd8-a144-c4c629341e42-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.081117 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.081097    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93debedb-47b6-4fd8-a144-c4c629341e42-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.093980 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.093945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93debedb-47b6-4fd8-a144-c4c629341e42-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.094361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.094343    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njszn\" (UniqueName: \"kubernetes.io/projected/93debedb-47b6-4fd8-a144-c4c629341e42-kube-api-access-njszn\") pod \"maas-default-gateway-openshift-default-78bb78ff78-4x7rq\" (UID: \"93debedb-47b6-4fd8-a144-c4c629341e42\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.165856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.165760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:21.451730 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.451699    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"]
Sep 17 14:17:21.454360 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:17:21.454327    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93debedb_47b6_4fd8_a144_c4c629341e42.slice/crio-50dc4a3565fe07025fa2e5c5f4965ee525c538cd147f9cfbb4f43a58256def2d WatchSource:0}: Error finding container 50dc4a3565fe07025fa2e5c5f4965ee525c538cd147f9cfbb4f43a58256def2d: Status 404 returned error can't find the container with id 50dc4a3565fe07025fa2e5c5f4965ee525c538cd147f9cfbb4f43a58256def2d
Sep 17 14:17:21.456458 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.456423    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:21.456554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.456498    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:21.456554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:21.456527    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:17:22.233049 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:22.233007    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" event={"ID":"93debedb-47b6-4fd8-a144-c4c629341e42","Type":"ContainerStarted","Data":"c7ae11d47e10ef4318cdd37a7d5dd594657ab834d9cdbee7d8fdfd98cefe190a"}
Sep 17 14:17:22.233478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:22.233065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" event={"ID":"93debedb-47b6-4fd8-a144-c4c629341e42","Type":"ContainerStarted","Data":"50dc4a3565fe07025fa2e5c5f4965ee525c538cd147f9cfbb4f43a58256def2d"}
Sep 17 14:17:22.259302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:22.259228    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" podStartSLOduration=2.259208293 podStartE2EDuration="2.259208293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:22.257080522 +0000 UTC m=+629.190585715" watchObservedRunningTime="2026-09-17 14:17:22.259208293 +0000 UTC m=+629.192713541"
Sep 17 14:17:23.166768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:23.166709    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:23.171927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:23.171891    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:23.237739 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:23.237701    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:23.238682 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:23.238662    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:17:23.497525 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:23.497418    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:23.631668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:23.631618    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:24.375038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.374974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:24.532682 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.532641    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:24.679424 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.679338    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-vgmp5"]
Sep 17 14:17:24.682036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.682016    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vgmp5"
Sep 17 14:17:24.685842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.685816    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-6svbb\""
Sep 17 14:17:24.702913 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.702890    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vgmp5"]
Sep 17 14:17:24.816592 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.816554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8t4v\" (UniqueName: \"kubernetes.io/projected/1e4f264d-7281-47e2-855b-9f4dd326325b-kube-api-access-p8t4v\") pod \"authorino-59c974bd48-vgmp5\" (UID: \"1e4f264d-7281-47e2-855b-9f4dd326325b\") " pod="kuadrant-system/authorino-59c974bd48-vgmp5"
Sep 17 14:17:24.918017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.917975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8t4v\" (UniqueName: \"kubernetes.io/projected/1e4f264d-7281-47e2-855b-9f4dd326325b-kube-api-access-p8t4v\") pod \"authorino-59c974bd48-vgmp5\" (UID: \"1e4f264d-7281-47e2-855b-9f4dd326325b\") " pod="kuadrant-system/authorino-59c974bd48-vgmp5"
Sep 17 14:17:24.937973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.937899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8t4v\" (UniqueName: \"kubernetes.io/projected/1e4f264d-7281-47e2-855b-9f4dd326325b-kube-api-access-p8t4v\") pod \"authorino-59c974bd48-vgmp5\" (UID: \"1e4f264d-7281-47e2-855b-9f4dd326325b\") " pod="kuadrant-system/authorino-59c974bd48-vgmp5"
Sep 17 14:17:24.991808 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:24.991769    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vgmp5"
Sep 17 14:17:25.270642 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:25.270611    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vgmp5"]
Sep 17 14:17:25.272838 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:17:25.272810    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e4f264d_7281_47e2_855b_9f4dd326325b.slice/crio-af3c4e0c1a62fad27e0ef130338bec31220e7f085aefa4bee84da05e4174c57a WatchSource:0}: Error finding container af3c4e0c1a62fad27e0ef130338bec31220e7f085aefa4bee84da05e4174c57a: Status 404 returned error can't find the container with id af3c4e0c1a62fad27e0ef130338bec31220e7f085aefa4bee84da05e4174c57a
Sep 17 14:17:26.255242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:26.255197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vgmp5" event={"ID":"1e4f264d-7281-47e2-855b-9f4dd326325b","Type":"ContainerStarted","Data":"af3c4e0c1a62fad27e0ef130338bec31220e7f085aefa4bee84da05e4174c57a"}
Sep 17 14:17:27.261122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:27.261076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vgmp5" event={"ID":"1e4f264d-7281-47e2-855b-9f4dd326325b","Type":"ContainerStarted","Data":"c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9"}
Sep 17 14:17:27.278680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:27.278617    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-vgmp5" podStartSLOduration=1.749628327 podStartE2EDuration="3.278597227s" totalImagesPullingTime="1.5289689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:27.278285471 +0000 UTC m=+634.211790665" watchObservedRunningTime="2026-09-17 14:17:27.278597227 +0000 UTC m=+634.212102420"
Sep 17 14:17:39.501114 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:39.501074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:39.588410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:39.588360    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:39.987482 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:39.987450    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:40.341696 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:40.341664    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:47.920308 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:47.920267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:47.997715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:47.997688    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:52.069135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:52.069101    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:52.200086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:52.200045    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:52.918884 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:52.918849    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fbcc6f98f-94vnx"]
Sep 17 14:17:52.921403 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:52.921382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:17:52.924378 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:52.924353    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 14:17:53.038308 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.038247    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c348f426-e106-4c38-8c8d-f7e0c609e99d-tls-cert\") pod \"authorino-6fbcc6f98f-94vnx\" (UID: \"c348f426-e106-4c38-8c8d-f7e0c609e99d\") " pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:17:53.038490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.038360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52p5f\" (UniqueName: \"kubernetes.io/projected/c348f426-e106-4c38-8c8d-f7e0c609e99d-kube-api-access-52p5f\") pod \"authorino-6fbcc6f98f-94vnx\" (UID: \"c348f426-e106-4c38-8c8d-f7e0c609e99d\") " pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:17:53.044706 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.044661    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fbcc6f98f-94vnx"]
Sep 17 14:17:53.139015 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.138972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c348f426-e106-4c38-8c8d-f7e0c609e99d-tls-cert\") pod \"authorino-6fbcc6f98f-94vnx\" (UID: \"c348f426-e106-4c38-8c8d-f7e0c609e99d\") " pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:17:53.139478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.139042    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52p5f\" (UniqueName: \"kubernetes.io/projected/c348f426-e106-4c38-8c8d-f7e0c609e99d-kube-api-access-52p5f\") pod \"authorino-6fbcc6f98f-94vnx\" (UID: \"c348f426-e106-4c38-8c8d-f7e0c609e99d\") " pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:17:53.141609 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.141587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c348f426-e106-4c38-8c8d-f7e0c609e99d-tls-cert\") pod \"authorino-6fbcc6f98f-94vnx\" (UID: \"c348f426-e106-4c38-8c8d-f7e0c609e99d\") " pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:17:53.162664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.162628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52p5f\" (UniqueName: \"kubernetes.io/projected/c348f426-e106-4c38-8c8d-f7e0c609e99d-kube-api-access-52p5f\") pod \"authorino-6fbcc6f98f-94vnx\" (UID: \"c348f426-e106-4c38-8c8d-f7e0c609e99d\") " pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:17:53.230470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.230370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:17:53.534199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:53.534171    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fbcc6f98f-94vnx"]
Sep 17 14:17:53.536110 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:17:53.536084    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc348f426_e106_4c38_8c8d_f7e0c609e99d.slice/crio-a67291c5b8943aef6ca85c5696ed52d7366da0f8a6e550d02377cf21efc5c4ee WatchSource:0}: Error finding container a67291c5b8943aef6ca85c5696ed52d7366da0f8a6e550d02377cf21efc5c4ee: Status 404 returned error can't find the container with id a67291c5b8943aef6ca85c5696ed52d7366da0f8a6e550d02377cf21efc5c4ee
Sep 17 14:17:54.384292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.384163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx" event={"ID":"c348f426-e106-4c38-8c8d-f7e0c609e99d","Type":"ContainerStarted","Data":"e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672"}
Sep 17 14:17:54.384292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.384213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx" event={"ID":"c348f426-e106-4c38-8c8d-f7e0c609e99d","Type":"ContainerStarted","Data":"a67291c5b8943aef6ca85c5696ed52d7366da0f8a6e550d02377cf21efc5c4ee"}
Sep 17 14:17:54.423000 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.422934    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx" podStartSLOduration=1.858006617 podStartE2EDuration="2.422916738s" totalImagesPullingTime="564.910121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:17:54.422008518 +0000 UTC m=+661.355513711" watchObservedRunningTime="2026-09-17 14:17:54.422916738 +0000 UTC m=+661.356421930"
Sep 17 14:17:54.497441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.497407    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vgmp5"]
Sep 17 14:17:54.497612 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.497585    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-vgmp5" podUID="1e4f264d-7281-47e2-855b-9f4dd326325b" containerName="authorino" containerID="cri-o://c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9" gracePeriod=30
Sep 17 14:17:54.744852 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.744824    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vgmp5"
Sep 17 14:17:54.856390 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.856345    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p8t4v\" (UniqueName: \"kubernetes.io/projected/1e4f264d-7281-47e2-855b-9f4dd326325b-kube-api-access-p8t4v\") pod \"1e4f264d-7281-47e2-855b-9f4dd326325b\" (UID: \"1e4f264d-7281-47e2-855b-9f4dd326325b\") "
Sep 17 14:17:54.858512 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.858476    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4f264d-7281-47e2-855b-9f4dd326325b-kube-api-access-p8t4v" (OuterVolumeSpecName: "kube-api-access-p8t4v") pod "1e4f264d-7281-47e2-855b-9f4dd326325b" (UID: "1e4f264d-7281-47e2-855b-9f4dd326325b"). InnerVolumeSpecName "kube-api-access-p8t4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:17:54.919338 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.919222    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:54.957038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:54.956995    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8t4v\" (UniqueName: \"kubernetes.io/projected/1e4f264d-7281-47e2-855b-9f4dd326325b-kube-api-access-p8t4v\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:17:55.071811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.071764    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:55.390824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.390784    2565 generic.go:356] "Generic (PLEG): container finished" podID="1e4f264d-7281-47e2-855b-9f4dd326325b" containerID="c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9" exitCode=0
Sep 17 14:17:55.391302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.390835    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-vgmp5"
Sep 17 14:17:55.391302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.390869    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vgmp5" event={"ID":"1e4f264d-7281-47e2-855b-9f4dd326325b","Type":"ContainerDied","Data":"c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9"}
Sep 17 14:17:55.391302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.390917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-vgmp5" event={"ID":"1e4f264d-7281-47e2-855b-9f4dd326325b","Type":"ContainerDied","Data":"af3c4e0c1a62fad27e0ef130338bec31220e7f085aefa4bee84da05e4174c57a"}
Sep 17 14:17:55.391302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.390934    2565 scope.go:117] "RemoveContainer" containerID="c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9"
Sep 17 14:17:55.400547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.400527    2565 scope.go:117] "RemoveContainer" containerID="c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9"
Sep 17 14:17:55.403536 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:17:55.403500    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9\": container with ID starting with c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9 not found: ID does not exist" containerID="c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9"
Sep 17 14:17:55.403690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.403547    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9"} err="failed to get container status \"c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9\": rpc error: code = NotFound desc = could not find container \"c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9\": container with ID starting with c817d9cbdcdb2c4584f92ce2b95a330336990e980372ad1b3c939768c920f7c9 not found: ID does not exist"
Sep 17 14:17:55.414729 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.414689    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vgmp5"]
Sep 17 14:17:55.419072 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.419040    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-vgmp5"]
Sep 17 14:17:55.519963 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:55.519930    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e4f264d-7281-47e2-855b-9f4dd326325b" path="/var/lib/kubelet/pods/1e4f264d-7281-47e2-855b-9f4dd326325b/volumes"
Sep 17 14:17:59.659478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:59.659446    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:59.782132 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:59.782103    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:17:59.944905 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:17:59.944824    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:00.216367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:00.216289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:03.790987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:03.790953    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:03.912960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:03.912927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:26.980440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:26.980405    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:27.083923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:27.083889    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:35.462172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:35.462134    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:35.608984 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:35.608950    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:37.101128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:37.101095    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:37.235505 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:37.235473    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:37.327551 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:37.327507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:37.675487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:37.675417    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:37.786399 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:37.786361    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:47.631960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:47.631925    2565 generic.go:356] "Generic (PLEG): container finished" podID="93debedb-47b6-4fd8-a144-c4c629341e42" containerID="c7ae11d47e10ef4318cdd37a7d5dd594657ab834d9cdbee7d8fdfd98cefe190a" exitCode=0
Sep 17 14:18:47.632376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:47.632000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" event={"ID":"93debedb-47b6-4fd8-a144-c4c629341e42","Type":"ContainerDied","Data":"c7ae11d47e10ef4318cdd37a7d5dd594657ab834d9cdbee7d8fdfd98cefe190a"}
Sep 17 14:18:47.632421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:47.632388    2565 scope.go:117] "RemoveContainer" containerID="c7ae11d47e10ef4318cdd37a7d5dd594657ab834d9cdbee7d8fdfd98cefe190a"
Sep 17 14:18:47.633416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:47.633391    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:47.633478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:47.633458    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:47.633515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:47.633488    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:18:48.064679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:48.064587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:48.198490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:48.198456    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:18:48.639070 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:48.639037    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" event={"ID":"93debedb-47b6-4fd8-a144-c4c629341e42","Type":"ContainerStarted","Data":"5444f23611464918db43ec12b20cbfe2d2bacb76477389e05641a4dce1cdb1b4"}
Sep 17 14:18:49.167277 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:49.167229    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:18:49.167708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:49.167685    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4x7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 17 14:18:49.167897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:49.167872    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" podUID="93debedb-47b6-4fd8-a144-c4c629341e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 17 14:18:50.166230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:50.166188    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4x7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 17 14:18:50.166859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:50.166825    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" podUID="93debedb-47b6-4fd8-a144-c4c629341e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 17 14:18:51.166239 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:51.166204    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4x7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 17 14:18:51.166845 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:51.166803    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" podUID="93debedb-47b6-4fd8-a144-c4c629341e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 17 14:18:52.166943 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:52.166907    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4x7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 17 14:18:52.167880 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:52.167497    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" podUID="93debedb-47b6-4fd8-a144-c4c629341e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 17 14:18:53.166361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:53.166324    2565 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-4x7rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Sep 17 14:18:53.166625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:53.166599    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq" podUID="93debedb-47b6-4fd8-a144-c4c629341e42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Sep 17 14:18:53.237893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:53.237864    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:18:54.170331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:54.170302    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:18:54.171120 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:18:54.171099    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-4x7rq"
Sep 17 14:19:09.940038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.940004    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"]
Sep 17 14:19:09.940527 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.940380    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e4f264d-7281-47e2-855b-9f4dd326325b" containerName="authorino"
Sep 17 14:19:09.940527 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.940392    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e4f264d-7281-47e2-855b-9f4dd326325b" containerName="authorino"
Sep 17 14:19:09.940527 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.940462    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e4f264d-7281-47e2-855b-9f4dd326325b" containerName="authorino"
Sep 17 14:19:09.943149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.943132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:09.946905 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.946885    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-pqctc\""
Sep 17 14:19:09.947537 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.947520    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 14:19:09.947603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.947538    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 14:19:09.947653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.947538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 17 14:19:09.961442 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:09.961420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"]
Sep 17 14:19:10.035570 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.035539    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:10.069622 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.069592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xxb\" (UniqueName: \"kubernetes.io/projected/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-kube-api-access-92xxb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.069740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.069632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.069740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.069659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.069740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.069701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.069740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.069717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.069902 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.069747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.170600 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.170571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xxb\" (UniqueName: \"kubernetes.io/projected/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-kube-api-access-92xxb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.170746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.170606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.170746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.170624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.170746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.170648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.170746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.170669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.170746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.170699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.171060 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.171035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.171135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.171049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.171135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.171122    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.173025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.173003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.173209 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.173192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.179409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.179390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xxb\" (UniqueName: \"kubernetes.io/projected/7ebba19f-bfe0-4f42-8c3d-fe2de60ce513-kube-api-access-92xxb\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx\" (UID: \"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.253297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.253214    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:10.551192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.551159    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"]
Sep 17 14:19:10.553870 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:19:10.553837    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebba19f_bfe0_4f42_8c3d_fe2de60ce513.slice/crio-6e50736125a1f7c12ddb2ee1cb77018f95746d163aecca1564edeed2b2e78a20 WatchSource:0}: Error finding container 6e50736125a1f7c12ddb2ee1cb77018f95746d163aecca1564edeed2b2e78a20: Status 404 returned error can't find the container with id 6e50736125a1f7c12ddb2ee1cb77018f95746d163aecca1564edeed2b2e78a20
Sep 17 14:19:10.752883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:10.752843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx" event={"ID":"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513","Type":"ContainerStarted","Data":"6e50736125a1f7c12ddb2ee1cb77018f95746d163aecca1564edeed2b2e78a20"}
Sep 17 14:19:11.759609 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:11.759569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx" event={"ID":"7ebba19f-bfe0-4f42-8c3d-fe2de60ce513","Type":"ContainerStarted","Data":"433fafe636174077c681ce2c5a40cf94c7ff4009d3dfc3f7a99bacc94ca08e57"}
Sep 17 14:19:11.814230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:11.814183    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx" podStartSLOduration=1.701762439 podStartE2EDuration="2.814167441s" totalImagesPullingTime="1.112405002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:11.812308614 +0000 UTC m=+738.745813806" watchObservedRunningTime="2026-09-17 14:19:11.814167441 +0000 UTC m=+738.747672632"
Sep 17 14:19:20.253985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:20.253949    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:20.254416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:20.254009    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:20.266523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:20.266499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:20.815024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:20.815001    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx"
Sep 17 14:19:25.179222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:25.179183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.062827 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.062795    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-588df94cbc-45qn6"]
Sep 17 14:19:29.067615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.067593    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.072627 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.072601    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 14:19:29.072627 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.072616    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 14:19:29.072897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.072601    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 14:19:29.072897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.072874    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 14:19:29.073511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.073484    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-45dp8\""
Sep 17 14:19:29.073511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.073486    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 14:19:29.073804 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.073773    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 14:19:29.074339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.074326    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 14:19:29.080463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.080442    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 14:19:29.088063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.088040    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588df94cbc-45qn6"]
Sep 17 14:19:29.170779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.170755    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.229025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.229004    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-oauth-config\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.229128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.229030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk6sx\" (UniqueName: \"kubernetes.io/projected/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-kube-api-access-wk6sx\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.229128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.229081    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-serving-cert\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.229128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.229096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-service-ca\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.229128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.229115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-config\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.229316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.229132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-oauth-serving-cert\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.229316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.229199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-trusted-ca-bundle\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.329539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.329521    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-oauth-config\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.329632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.329547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk6sx\" (UniqueName: \"kubernetes.io/projected/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-kube-api-access-wk6sx\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.329632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.329590    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-serving-cert\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.329632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.329613    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-service-ca\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.329778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.329642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-config\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.329778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.329671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-oauth-serving-cert\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.329778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.329703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-trusted-ca-bundle\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.330456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.330436    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-service-ca\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.330557 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.330499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-oauth-serving-cert\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.330606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.330563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-config\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.330892 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.330868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-trusted-ca-bundle\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.332207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.332189    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-serving-cert\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.332318 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.332229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-console-oauth-config\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.339960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.339936    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk6sx\" (UniqueName: \"kubernetes.io/projected/d79e67f7-6318-46da-8fe0-fc9a5cf5948a-kube-api-access-wk6sx\") pod \"console-588df94cbc-45qn6\" (UID: \"d79e67f7-6318-46da-8fe0-fc9a5cf5948a\") " pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.347401 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.347371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.377292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.377269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:29.436506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.436478    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.657404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.657376    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588df94cbc-45qn6"]
Sep 17 14:19:29.659432 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:19:29.659404    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd79e67f7_6318_46da_8fe0_fc9a5cf5948a.slice/crio-66a8ceca7c70cd8b94fa427af35bbb2bb6b9d818d7c739dfd53cd7d1c2e185d4 WatchSource:0}: Error finding container 66a8ceca7c70cd8b94fa427af35bbb2bb6b9d818d7c739dfd53cd7d1c2e185d4: Status 404 returned error can't find the container with id 66a8ceca7c70cd8b94fa427af35bbb2bb6b9d818d7c739dfd53cd7d1c2e185d4
Sep 17 14:19:29.832434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.832394    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:29.847047 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.847012    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588df94cbc-45qn6" event={"ID":"d79e67f7-6318-46da-8fe0-fc9a5cf5948a","Type":"ContainerStarted","Data":"ac9bb843ef5b63f071dbdeb0dfec1ecc80bbc7e8d866b968d3b9b98ced3f8de7"}
Sep 17 14:19:29.847172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.847055    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588df94cbc-45qn6" event={"ID":"d79e67f7-6318-46da-8fe0-fc9a5cf5948a","Type":"ContainerStarted","Data":"66a8ceca7c70cd8b94fa427af35bbb2bb6b9d818d7c739dfd53cd7d1c2e185d4"}
Sep 17 14:19:29.868214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.868170    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-588df94cbc-45qn6" podStartSLOduration=0.868155431 podStartE2EDuration="868.155431ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:29.865967401 +0000 UTC m=+756.799472593" watchObservedRunningTime="2026-09-17 14:19:29.868155431 +0000 UTC m=+756.801660622"
Sep 17 14:19:29.932670 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:29.932603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:33.748695 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.748657    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"]
Sep 17 14:19:33.752585 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.752565    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.755488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.755465    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 17 14:19:33.765820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.765794    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"]
Sep 17 14:19:33.837558 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.837525    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:33.865633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.865606    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.865760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.865637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.865760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.865692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.865895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.865772    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lgj2\" (UniqueName: \"kubernetes.io/projected/b1fe51fb-e6fb-4048-93e6-88d1d98173af-kube-api-access-9lgj2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.865895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.865799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1fe51fb-e6fb-4048-93e6-88d1d98173af-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.865895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.865832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.966894 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.966868    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lgj2\" (UniqueName: \"kubernetes.io/projected/b1fe51fb-e6fb-4048-93e6-88d1d98173af-kube-api-access-9lgj2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.967005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.966899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1fe51fb-e6fb-4048-93e6-88d1d98173af-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.967005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.966928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.967005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.966949    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.967005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.966990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.967181 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.967054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.967380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.967359    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.967470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.967410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.967527 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.967472    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.969057 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.969037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1fe51fb-e6fb-4048-93e6-88d1d98173af-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.969485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.969466    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1fe51fb-e6fb-4048-93e6-88d1d98173af-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:33.976969 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:33.976943    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lgj2\" (UniqueName: \"kubernetes.io/projected/b1fe51fb-e6fb-4048-93e6-88d1d98173af-kube-api-access-9lgj2\") pod \"e2e-distinct-simulated-kserve-59975d69f9-kpms6\" (UID: \"b1fe51fb-e6fb-4048-93e6-88d1d98173af\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:34.062885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:34.062819    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:34.338024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:34.337997    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"]
Sep 17 14:19:34.340218 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:19:34.340177    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1fe51fb_e6fb_4048_93e6_88d1d98173af.slice/crio-bdfe17d45f6b551eed5c274dad56ff062ffe3e1daf129b2a90dd812840cec529 WatchSource:0}: Error finding container bdfe17d45f6b551eed5c274dad56ff062ffe3e1daf129b2a90dd812840cec529: Status 404 returned error can't find the container with id bdfe17d45f6b551eed5c274dad56ff062ffe3e1daf129b2a90dd812840cec529
Sep 17 14:19:34.344540 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:34.344517    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 14:19:34.873894 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:34.873805    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6" event={"ID":"b1fe51fb-e6fb-4048-93e6-88d1d98173af","Type":"ContainerStarted","Data":"de1a45eb0b29249f6ec6bdf05545f3009527162e6b7c7e441bdb714a8ee76415"}
Sep 17 14:19:34.873894 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:34.873864    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6" event={"ID":"b1fe51fb-e6fb-4048-93e6-88d1d98173af","Type":"ContainerStarted","Data":"bdfe17d45f6b551eed5c274dad56ff062ffe3e1daf129b2a90dd812840cec529"}
Sep 17 14:19:34.898547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:34.898485    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6" podStartSLOduration=1.667471924 podStartE2EDuration="1.898466198s" totalImagesPullingTime="230.994274ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:19:34.896792881 +0000 UTC m=+761.830298115" watchObservedRunningTime="2026-09-17 14:19:34.898466198 +0000 UTC m=+761.831971398"
Sep 17 14:19:39.377767 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:39.377731    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:39.378148 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:39.377801    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:39.382879 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:39.382851    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:39.902363 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:39.902330    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-588df94cbc-45qn6"
Sep 17 14:19:40.091136 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:40.091096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:40.192229 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:40.192130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:40.292689 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:40.292644    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:44.063648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:44.063614    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:44.064115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:44.063668    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:44.076520 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:44.076486    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:19:44.814827 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:44.814795    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:19:44.934778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:19:44.934744    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-kpms6"
Sep 17 14:20:06.166822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:06.166781    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:10.275369 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:10.275333    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:23.104842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:23.104752    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:31.155607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.155568    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"]
Sep 17 14:20:31.159581 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.159563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.164187 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.164166    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 14:20:31.203222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.203190    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"]
Sep 17 14:20:31.221348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.221318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5742r\" (UniqueName: \"kubernetes.io/projected/e101f8ab-94d0-41d6-b81a-39cb52381ddc-kube-api-access-5742r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.221521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.221352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.221521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.221376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.221521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.221415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e101f8ab-94d0-41d6-b81a-39cb52381ddc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.221521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.221433    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.221521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.221448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.255199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.255158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:31.322392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5742r\" (UniqueName: \"kubernetes.io/projected/e101f8ab-94d0-41d6-b81a-39cb52381ddc-kube-api-access-5742r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.322392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.322638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.322638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e101f8ab-94d0-41d6-b81a-39cb52381ddc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.322638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.322638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.322958 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322922    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.323097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322975    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.323097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.322973    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.324754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.324732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e101f8ab-94d0-41d6-b81a-39cb52381ddc-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.324893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.324874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e101f8ab-94d0-41d6-b81a-39cb52381ddc-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.337591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.337563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5742r\" (UniqueName: \"kubernetes.io/projected/e101f8ab-94d0-41d6-b81a-39cb52381ddc-kube-api-access-5742r\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-zxbv8\" (UID: \"e101f8ab-94d0-41d6-b81a-39cb52381ddc\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.469730 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.469639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:31.753867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:31.753844    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"]
Sep 17 14:20:31.757145 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:20:31.757109    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode101f8ab_94d0_41d6_b81a_39cb52381ddc.slice/crio-2a3dd3345bd20873026f924cbb9e2c8ecafb68ba474eb92ae7d55739fc9cc998 WatchSource:0}: Error finding container 2a3dd3345bd20873026f924cbb9e2c8ecafb68ba474eb92ae7d55739fc9cc998: Status 404 returned error can't find the container with id 2a3dd3345bd20873026f924cbb9e2c8ecafb68ba474eb92ae7d55739fc9cc998
Sep 17 14:20:32.164834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:32.164802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8" event={"ID":"e101f8ab-94d0-41d6-b81a-39cb52381ddc","Type":"ContainerStarted","Data":"359a145de9bc97871b0f8b771609c15f730650da782bc8b89412eea67f8de764"}
Sep 17 14:20:32.165336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:32.164854    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8" event={"ID":"e101f8ab-94d0-41d6-b81a-39cb52381ddc","Type":"ContainerStarted","Data":"2a3dd3345bd20873026f924cbb9e2c8ecafb68ba474eb92ae7d55739fc9cc998"}
Sep 17 14:20:32.194971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:32.194924    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8" podStartSLOduration=0.958807328 podStartE2EDuration="1.194909002s" totalImagesPullingTime="236.101674ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:20:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:20:32.192270505 +0000 UTC m=+819.125775703" watchObservedRunningTime="2026-09-17 14:20:32.194909002 +0000 UTC m=+819.128414193"
Sep 17 14:20:34.546000 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:34.545966    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:20:41.470831 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:41.470778    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:41.471434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:41.470892    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:41.484575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:41.484544    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:20:42.226616 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:20:42.226586    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-zxbv8"
Sep 17 14:21:28.496424 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.496389    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fbcc6f98f-94vnx"]
Sep 17 14:21:28.497002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.496557    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx" podUID="c348f426-e106-4c38-8c8d-f7e0c609e99d" containerName="authorino" containerID="cri-o://e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672" gracePeriod=30
Sep 17 14:21:28.747922 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.747862    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:21:28.848789 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.848761    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52p5f\" (UniqueName: \"kubernetes.io/projected/c348f426-e106-4c38-8c8d-f7e0c609e99d-kube-api-access-52p5f\") pod \"c348f426-e106-4c38-8c8d-f7e0c609e99d\" (UID: \"c348f426-e106-4c38-8c8d-f7e0c609e99d\") "
Sep 17 14:21:28.848938 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.848905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c348f426-e106-4c38-8c8d-f7e0c609e99d-tls-cert\") pod \"c348f426-e106-4c38-8c8d-f7e0c609e99d\" (UID: \"c348f426-e106-4c38-8c8d-f7e0c609e99d\") "
Sep 17 14:21:28.851054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.851024    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c348f426-e106-4c38-8c8d-f7e0c609e99d-kube-api-access-52p5f" (OuterVolumeSpecName: "kube-api-access-52p5f") pod "c348f426-e106-4c38-8c8d-f7e0c609e99d" (UID: "c348f426-e106-4c38-8c8d-f7e0c609e99d"). InnerVolumeSpecName "kube-api-access-52p5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:21:28.858687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.858660    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c348f426-e106-4c38-8c8d-f7e0c609e99d-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c348f426-e106-4c38-8c8d-f7e0c609e99d" (UID: "c348f426-e106-4c38-8c8d-f7e0c609e99d"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:21:28.950664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.950634    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52p5f\" (UniqueName: \"kubernetes.io/projected/c348f426-e106-4c38-8c8d-f7e0c609e99d-kube-api-access-52p5f\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:21:28.950770 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:28.950664    2565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c348f426-e106-4c38-8c8d-f7e0c609e99d-tls-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:21:29.443353 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.443321    2565 generic.go:356] "Generic (PLEG): container finished" podID="c348f426-e106-4c38-8c8d-f7e0c609e99d" containerID="e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672" exitCode=0
Sep 17 14:21:29.443531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.443361    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx"
Sep 17 14:21:29.443531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.443407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx" event={"ID":"c348f426-e106-4c38-8c8d-f7e0c609e99d","Type":"ContainerDied","Data":"e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672"}
Sep 17 14:21:29.443531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.443451    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fbcc6f98f-94vnx" event={"ID":"c348f426-e106-4c38-8c8d-f7e0c609e99d","Type":"ContainerDied","Data":"a67291c5b8943aef6ca85c5696ed52d7366da0f8a6e550d02377cf21efc5c4ee"}
Sep 17 14:21:29.443531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.443469    2565 scope.go:117] "RemoveContainer" containerID="e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672"
Sep 17 14:21:29.453090 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.453074    2565 scope.go:117] "RemoveContainer" containerID="e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672"
Sep 17 14:21:29.453369 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:21:29.453351    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672\": container with ID starting with e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672 not found: ID does not exist" containerID="e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672"
Sep 17 14:21:29.453455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.453376    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672"} err="failed to get container status \"e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672\": rpc error: code = NotFound desc = could not find container \"e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672\": container with ID starting with e0b9f0bcc1b06bc5f8775ca3eab51f56f938e14d231c6fe68b032c5fbcf08672 not found: ID does not exist"
Sep 17 14:21:29.499819 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.499797    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fbcc6f98f-94vnx"]
Sep 17 14:21:29.501725 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.501702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:29.518975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.518957    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6fbcc6f98f-94vnx"]
Sep 17 14:21:29.616113 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:29.616087    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:31.034431 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:31.034380    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:31.259353 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:31.259275    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:31.519155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:31.519127    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c348f426-e106-4c38-8c8d-f7e0c609e99d" path="/var/lib/kubelet/pods/c348f426-e106-4c38-8c8d-f7e0c609e99d/volumes"
Sep 17 14:21:49.423459 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.423361    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:49.570890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.570859    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:49.683010 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.682941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:49.798628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.798588    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:49.939222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.939152    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"]
Sep 17 14:21:49.939588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.939572    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c348f426-e106-4c38-8c8d-f7e0c609e99d" containerName="authorino"
Sep 17 14:21:49.939633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.939591    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c348f426-e106-4c38-8c8d-f7e0c609e99d" containerName="authorino"
Sep 17 14:21:49.939673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.939656    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c348f426-e106-4c38-8c8d-f7e0c609e99d" containerName="authorino"
Sep 17 14:21:49.942758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.942740    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:49.946500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.946480    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2e1b82-openshift-default-dockercfg-httmw\""
Sep 17 14:21:49.964508 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:49.964469    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"]
Sep 17 14:21:50.025997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.025950    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-credential-socket\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.026169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.026033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-podinfo\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.026169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.026088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-socket\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.026169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.026120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jfn\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-kube-api-access-69jfn\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.026169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.026146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-certs\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.026433 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.026353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-data\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.026433 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.026398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-envoy\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.026544 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.026454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istiod-ca-cert\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.026544 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.026513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-token\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.054805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.054770    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"]
Sep 17 14:21:50.058749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.058726    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.060975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.060955    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8b50f2-openshift-default-dockercfg-j75vs\""
Sep 17 14:21:50.072742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.072715    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"]
Sep 17 14:21:50.127666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-credential-socket\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.127666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127671    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-envoy\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.127910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-podinfo\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.127910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istiod-ca-cert\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.127910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127777    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-socket\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.127910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jfn\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-kube-api-access-69jfn\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.127910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-certs\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.127910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-podinfo\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-credential-socket\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.127952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-data\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-socket\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-envoy\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-token\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-credential-socket\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-certs\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128124    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-socket\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istiod-ca-cert\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldrcj\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-kube-api-access-ldrcj\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.128298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-data\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.128877 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128308    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-certs\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.128877 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-token\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.128877 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-data\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.129047 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.128896    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istiod-ca-cert\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.130416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.130389    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-envoy\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.130573 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.130555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-podinfo\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.138039 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.138014    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:50.147889 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.147863    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-token\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.148022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.148002    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jfn\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-kube-api-access-69jfn\") pod \"e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") " pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.162374 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.162352    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"]
Sep 17 14:21:50.166913 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.166897    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.169886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.169864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-080801-openshift-default-dockercfg-b5bgg\""
Sep 17 14:21:50.181044 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.181021    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"]
Sep 17 14:21:50.229854 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.229762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-envoy\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.229854 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.229831    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-envoy\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230068 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.229882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-certs\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.230068 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.229909    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-data\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.230068 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.229970    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istiod-ca-cert\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230068 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-podinfo\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.230068 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxqqq\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-kube-api-access-pxqqq\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.230350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istiod-ca-cert\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.230350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230152    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-credential-socket\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.230350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-podinfo\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-credential-socket\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230354    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-socket\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230390    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-token\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-socket\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.230611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-certs\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230486    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldrcj\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-kube-api-access-ldrcj\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-data\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.230611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-token\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.230611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230564    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-credential-socket\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.231002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.230704    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istiod-ca-cert\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.231123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.231100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-certs\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.231172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.231153    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-data\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.231330 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.231313    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-socket\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.232608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.232590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-envoy\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.232856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.232836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-podinfo\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.236656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.236629    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:50.246427 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.246380    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-token\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.246545 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.246520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldrcj\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-kube-api-access-ldrcj\") pod \"e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") " pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.254276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.254007    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:50.332130 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-token\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332293 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-envoy\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332293 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-certs\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332293 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332269    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-data\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332476 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-podinfo\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332476 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxqqq\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-kube-api-access-pxqqq\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332476 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332389    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istiod-ca-cert\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332476 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332428    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-credential-socket\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-socket\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.332934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332907    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-certs\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.333009 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.332994    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-credential-socket\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.333307 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.333223    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-socket\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.333425 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.333378    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istiod-ca-cert\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.333425 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.333415    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-data\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.335002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.334983    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-podinfo\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.335117 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.335102    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-envoy\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.350412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.350385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-token\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.351362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.351339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxqqq\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-kube-api-access-pxqqq\") pod \"e2e-worker-w0-080801-openshift-default-84855b7986-dx695\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") " pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.369639 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.369619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:50.477769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.477738    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:50.533307 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.533157    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:50.559606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.558457    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"]
Sep 17 14:21:50.564889 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.564843    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.565024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.564937    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.565024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.564982    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.617937 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.617916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:50.650392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.650177    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"]
Sep 17 14:21:50.652882 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:21:50.651508    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6301c8c7_7838_4cc3_b149_8e1c506e3da7.slice/crio-21ba67538595b006560c63c1c389bba4aa3fbdcf1618302030ae87cf8e1eaae2 WatchSource:0}: Error finding container 21ba67538595b006560c63c1c389bba4aa3fbdcf1618302030ae87cf8e1eaae2: Status 404 returned error can't find the container with id 21ba67538595b006560c63c1c389bba4aa3fbdcf1618302030ae87cf8e1eaae2
Sep 17 14:21:50.654762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.654727    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.654871 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.654805    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.654871 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.654846    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.773901 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.773854    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"]
Sep 17 14:21:50.777390 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:21:50.777360    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309c5898_acbe_4ad6_ba82_c3a9c51c26c3.slice/crio-64880d299304641246bb8313315ff6641bfb2bd465ae69c55ecbecd430ecc113 WatchSource:0}: Error finding container 64880d299304641246bb8313315ff6641bfb2bd465ae69c55ecbecd430ecc113: Status 404 returned error can't find the container with id 64880d299304641246bb8313315ff6641bfb2bd465ae69c55ecbecd430ecc113
Sep 17 14:21:50.780033 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.780002    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.780133 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.780064    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.780133 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.780098    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:50.806693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:50.806670    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:51.176994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.176961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:51.283461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.283430    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:51.556510 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.556419    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" event={"ID":"309c5898-acbe-4ad6-ba82-c3a9c51c26c3","Type":"ContainerStarted","Data":"c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712"}
Sep 17 14:21:51.556510 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.556475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" event={"ID":"309c5898-acbe-4ad6-ba82-c3a9c51c26c3","Type":"ContainerStarted","Data":"64880d299304641246bb8313315ff6641bfb2bd465ae69c55ecbecd430ecc113"}
Sep 17 14:21:51.558073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.558040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" event={"ID":"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb","Type":"ContainerStarted","Data":"0e667d70396575a36ef701bb8c71517c551677509ab9716198d3f002949e2c16"}
Sep 17 14:21:51.558225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.558088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" event={"ID":"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb","Type":"ContainerStarted","Data":"b6408e1a8fecef314bdfc11f5f7fea654d0bb220043e3032112d65fbeb9078fa"}
Sep 17 14:21:51.559620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.559597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" event={"ID":"6301c8c7-7838-4cc3-b149-8e1c506e3da7","Type":"ContainerStarted","Data":"1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f"}
Sep 17 14:21:51.559704 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.559636    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" event={"ID":"6301c8c7-7838-4cc3-b149-8e1c506e3da7","Type":"ContainerStarted","Data":"21ba67538595b006560c63c1c389bba4aa3fbdcf1618302030ae87cf8e1eaae2"}
Sep 17 14:21:51.568263 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.568229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:51.587150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.587107    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podStartSLOduration=1.5870949460000001 podStartE2EDuration="1.587094946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:51.584495025 +0000 UTC m=+898.518000211" watchObservedRunningTime="2026-09-17 14:21:51.587094946 +0000 UTC m=+898.520600132"
Sep 17 14:21:51.609441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.609394    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podStartSLOduration=1.609380656 podStartE2EDuration="1.609380656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:51.608901214 +0000 UTC m=+898.542406430" watchObservedRunningTime="2026-09-17 14:21:51.609380656 +0000 UTC m=+898.542885848"
Sep 17 14:21:51.632623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:51.632577    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podStartSLOduration=2.632558606 podStartE2EDuration="2.632558606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:51.631230114 +0000 UTC m=+898.564735321" watchObservedRunningTime="2026-09-17 14:21:51.632558606 +0000 UTC m=+898.566063801"
Sep 17 14:21:52.255155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.255116    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:52.260533 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.260506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:52.369950 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.369904    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:52.375678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.375652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:52.478067 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.478040    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:52.482859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.482837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:52.565506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.565420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:52.565506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.565491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:52.565934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.565516    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:52.566182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.566154    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:21:52.566322 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.566223    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:21:52.566541 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:52.566521    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:21:53.490222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:53.490192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:21:53.491794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:53.491773    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:21:53.495553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:53.495531    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:21:53.497297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:53.497276    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:21:53.570779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:53.570752    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:53.710798 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:53.710759    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:53.933342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:53.933311    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:54.043966 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:54.043935    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:54.865619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:54.865581    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:55.055313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:55.055216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:55.247271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:55.247152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:55.463831 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:55.463798    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:55.567696 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:55.567616    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:56.063323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:56.063286    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:56.144485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:56.144456    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:56.267867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:56.267834    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:56.720858 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:56.720825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:56.811219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:56.811179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:57.388978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:57.388942    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:57.546931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:57.546901    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:57.644985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:57.644906    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:57.888961 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:57.888915    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:58.125970 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.125922    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:58.368507 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.368475    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"]
Sep 17 14:21:58.372005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.371983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.375015 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.374994    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-86c464-openshift-default-dockercfg-87vf9\""
Sep 17 14:21:58.389426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.389399    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"]
Sep 17 14:21:58.425667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425643    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-certs\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.425754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-socket\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.425754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istiod-ca-cert\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.425754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-podinfo\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.425868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-credential-socket\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.425868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-envoy\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.425868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-token\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.425868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-data\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.425997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.425900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsbg\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-kube-api-access-zfsbg\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527432 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-socket\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istiod-ca-cert\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-podinfo\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-credential-socket\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-envoy\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-token\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527715    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-data\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsbg\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-kube-api-access-zfsbg\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.527940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-socket\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.528102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.527948    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-certs\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.528102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.528033    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-credential-socket\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.528208 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.528186    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-data\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.528296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.528278    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-certs\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.528344 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.528299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istiod-ca-cert\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.530395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.530372    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-envoy\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.530596 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.530579    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-podinfo\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.542701 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.542677    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsbg\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-kube-api-access-zfsbg\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.542873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.542854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-token\") pod \"e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") " pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.692580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.692507    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:21:58.983919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.983895    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"]
Sep 17 14:21:58.985191 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:21:58.985164    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0ec2bc_d7f5_4743_9558_065b0336f86b.slice/crio-808d269397db4590a5e4355fc67ffa1f335877c8062ebb74362ae26494afbcf6 WatchSource:0}: Error finding container 808d269397db4590a5e4355fc67ffa1f335877c8062ebb74362ae26494afbcf6: Status 404 returned error can't find the container with id 808d269397db4590a5e4355fc67ffa1f335877c8062ebb74362ae26494afbcf6
Sep 17 14:21:58.987345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.987316    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:58.987391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.987377    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:58.987436 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:58.987406    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:21:59.386666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:59.386626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:21:59.601139 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:59.601108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" event={"ID":"7f0ec2bc-d7f5-4743-9558-065b0336f86b","Type":"ContainerStarted","Data":"a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b"}
Sep 17 14:21:59.601493 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:59.601158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" event={"ID":"7f0ec2bc-d7f5-4743-9558-065b0336f86b","Type":"ContainerStarted","Data":"808d269397db4590a5e4355fc67ffa1f335877c8062ebb74362ae26494afbcf6"}
Sep 17 14:21:59.630436 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:59.630391    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" podStartSLOduration=1.630376915 podStartE2EDuration="1.630376915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:21:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:21:59.627758402 +0000 UTC m=+906.561263595" watchObservedRunningTime="2026-09-17 14:21:59.630376915 +0000 UTC m=+906.563882107"
Sep 17 14:21:59.693395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:21:59.693329    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:22:00.557524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:00.557488    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:00.656327 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:00.656292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:00.698933 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:00.698909    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:22:00.789588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:00.789557    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:01.057898 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.057855    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"]
Sep 17 14:22:01.061798 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.061775    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.065105 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.065082    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a48a4ae7-openshift-default-dockercfg-cwjcr\""
Sep 17 14:22:01.080188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.080166    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"]
Sep 17 14:22:01.108963 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.108939    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:01.154621 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.154589    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-istio-token\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.154890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.154862    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-socket\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.155011 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.154995    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crxfm\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-kube-api-access-crxfm\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.155135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.155119    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-credential-socket\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.155280 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.155246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d732a81-709d-46bb-9832-75dfd05875bb-istio-podinfo\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.155404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.155389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d732a81-709d-46bb-9832-75dfd05875bb-istiod-ca-cert\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.155547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.155531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-envoy\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.155658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.155641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-certs\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.155800 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.155781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-data\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257237    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-envoy\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-certs\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257538 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-data\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257538 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-istio-token\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257538 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-socket\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257538 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crxfm\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-kube-api-access-crxfm\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-credential-socket\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257585    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d732a81-709d-46bb-9832-75dfd05875bb-istio-podinfo\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d732a81-709d-46bb-9832-75dfd05875bb-istiod-ca-cert\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257762    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-certs\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.257941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.257868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-socket\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.258063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.258042    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-credential-socket\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.258132 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.258092    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-data\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.258429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.258408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d732a81-709d-46bb-9832-75dfd05875bb-istiod-ca-cert\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.259595 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.259576    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-envoy\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.260210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.260191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d732a81-709d-46bb-9832-75dfd05875bb-istio-podinfo\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.273325 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.273296    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-istio-token\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.275824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.275803    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crxfm\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-kube-api-access-crxfm\") pod \"e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") " pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.372455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.372433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:01.560226 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.560186    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:01.610934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.610898    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:22:01.611812 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.611793    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:22:01.716748 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.716719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"]
Sep 17 14:22:01.717222 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:22:01.717193    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d732a81_709d_46bb_9832_75dfd05875bb.slice/crio-c0f85dc14d0e62e0e8f4dca67e0715dbc6a2d52e40657f40ad5929e9d1d9b94f WatchSource:0}: Error finding container c0f85dc14d0e62e0e8f4dca67e0715dbc6a2d52e40657f40ad5929e9d1d9b94f: Status 404 returned error can't find the container with id c0f85dc14d0e62e0e8f4dca67e0715dbc6a2d52e40657f40ad5929e9d1d9b94f
Sep 17 14:22:01.719178 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.719142    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:01.719290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.719218    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:01.719290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.719281    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:01.927648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:01.927615    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:02.077861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:02.077831    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:02.617471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:02.617437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" event={"ID":"8d732a81-709d-46bb-9832-75dfd05875bb","Type":"ContainerStarted","Data":"871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc"}
Sep 17 14:22:02.617663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:02.617490    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" event={"ID":"8d732a81-709d-46bb-9832-75dfd05875bb","Type":"ContainerStarted","Data":"c0f85dc14d0e62e0e8f4dca67e0715dbc6a2d52e40657f40ad5929e9d1d9b94f"}
Sep 17 14:22:02.641266 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:02.641205    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podStartSLOduration=1.641191026 podStartE2EDuration="1.641191026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:02.640031948 +0000 UTC m=+909.573537176" watchObservedRunningTime="2026-09-17 14:22:02.641191026 +0000 UTC m=+909.574696218"
Sep 17 14:22:03.373286 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:03.373222    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:03.384959 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:03.384922    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:03.430745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:03.430674    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:03.627805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:03.627704    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:03.630283 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:03.630168    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:04.278801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:04.278745    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:04.465948 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:04.465902    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:05.614789 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:05.614748    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:06.715116 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:06.715065    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:06.808793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:06.808687    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:07.256882 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:07.256841    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:07.349615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:07.349569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:07.818594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:07.818522    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:07.928471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:07.928400    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:08.033765 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.033710    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:08.320945 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.320870    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:08.451522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.451487    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:08.470466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.470434    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"]
Sep 17 14:22:08.477513 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.477484    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:08.481621 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.481581    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a48a4ae7\""
Sep 17 14:22:08.496945 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.496916    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"]
Sep 17 14:22:08.583303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.581501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tf9tt\" (UniqueName: \"kubernetes.io/projected/0419a270-844e-4a9e-8c44-b4abb8d160dd-kube-api-access-tf9tt\") pod \"payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9\" (UID: \"0419a270-844e-4a9e-8c44-b4abb8d160dd\") " pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:08.583303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.581561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0419a270-844e-4a9e-8c44-b4abb8d160dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9\" (UID: \"0419a270-844e-4a9e-8c44-b4abb8d160dd\") " pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:08.683275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.682556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tf9tt\" (UniqueName: \"kubernetes.io/projected/0419a270-844e-4a9e-8c44-b4abb8d160dd-kube-api-access-tf9tt\") pod \"payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9\" (UID: \"0419a270-844e-4a9e-8c44-b4abb8d160dd\") " pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:08.683275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.682612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0419a270-844e-4a9e-8c44-b4abb8d160dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9\" (UID: \"0419a270-844e-4a9e-8c44-b4abb8d160dd\") " pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:08.683455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.683327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0419a270-844e-4a9e-8c44-b4abb8d160dd-plugins-config-volume\") pod \"payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9\" (UID: \"0419a270-844e-4a9e-8c44-b4abb8d160dd\") " pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:08.692318 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.692292    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tf9tt\" (UniqueName: \"kubernetes.io/projected/0419a270-844e-4a9e-8c44-b4abb8d160dd-kube-api-access-tf9tt\") pod \"payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9\" (UID: \"0419a270-844e-4a9e-8c44-b4abb8d160dd\") " pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:08.795851 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.795822    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:08.824016 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:08.823989    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:09.036779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:09.036740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:09.135217 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:09.135189    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"]
Sep 17 14:22:09.137031 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:22:09.136999    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0419a270_844e_4a9e_8c44_b4abb8d160dd.slice/crio-560bfed2a0376204da9692caa4447af61af670ffc88718e67aa71077ef421d8e WatchSource:0}: Error finding container 560bfed2a0376204da9692caa4447af61af670ffc88718e67aa71077ef421d8e: Status 404 returned error can't find the container with id 560bfed2a0376204da9692caa4447af61af670ffc88718e67aa71077ef421d8e
Sep 17 14:22:09.244099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:09.244061    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:09.383833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:09.383744    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:09.487501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:09.487448    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:09.678454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:09.678311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9" event={"ID":"0419a270-844e-4a9e-8c44-b4abb8d160dd","Type":"ContainerStarted","Data":"560bfed2a0376204da9692caa4447af61af670ffc88718e67aa71077ef421d8e"}
Sep 17 14:22:10.764807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:10.764718    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:11.976048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:11.976007    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:12.091703 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:12.091633    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:12.206490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:12.206450    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:13.005807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:13.005765    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:13.485608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:13.485554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:13.946555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:13.589414    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:13.946555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:13.709234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9" event={"ID":"0419a270-844e-4a9e-8c44-b4abb8d160dd","Type":"ContainerStarted","Data":"10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce"}
Sep 17 14:22:13.946555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:13.709323    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:13.946555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:13.742721    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9" podStartSLOduration=1.984223014 podStartE2EDuration="5.742706394s" totalImagesPullingTime="3.75848338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:13.740333897 +0000 UTC m=+920.673839103" watchObservedRunningTime="2026-09-17 14:22:13.742706394 +0000 UTC m=+920.676211635"
Sep 17 14:22:13.946555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:13.836355    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:13.948220 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:13.946739    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:14.146189 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.146158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:14.335271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.331807    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"]
Sep 17 14:22:14.344291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.343358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.352016 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.351325    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-28f011-openshift-default-dockercfg-svnc8\""
Sep 17 14:22:14.383083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.383016    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"]
Sep 17 14:22:14.422410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.422381    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:14.453515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453484    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-credential-socket\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.453619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-socket\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.453619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453586    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-podinfo\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.453737 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-data\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.453808 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab092f48-a46e-498d-9f3c-369cc313e5c5-istiod-ca-cert\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.453872 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-certs\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.453929 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-token\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.453929 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453894    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-envoy\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.453929 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.453922    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrrq\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-kube-api-access-dlrrq\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.555598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-credential-socket\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.555598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-socket\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.555814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-podinfo\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.555814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-data\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.555814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab092f48-a46e-498d-9f3c-369cc313e5c5-istiod-ca-cert\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.555814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-certs\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.556027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-token\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.556027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-envoy\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.556027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrrq\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-kube-api-access-dlrrq\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.556027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-credential-socket\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.556027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.555959    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-socket\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.556628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.556368    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-certs\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.556628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.556625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-data\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.556783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.556658    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab092f48-a46e-498d-9f3c-369cc313e5c5-istiod-ca-cert\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.558554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.558531    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-envoy\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.558728 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.558714    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-podinfo\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.565699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.565678    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:14.567626 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.567604    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-token\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.574270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.574233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrrq\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-kube-api-access-dlrrq\") pod \"e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") " pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.662478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.662453    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:14.721665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.721179    2565 generic.go:356] "Generic (PLEG): container finished" podID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerID="c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712" exitCode=0
Sep 17 14:22:14.721665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.721286    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" event={"ID":"309c5898-acbe-4ad6-ba82-c3a9c51c26c3","Type":"ContainerDied","Data":"c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712"}
Sep 17 14:22:14.722662 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.722038    2565 scope.go:117] "RemoveContainer" containerID="c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712"
Sep 17 14:22:14.724435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.724190    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.724435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.724314    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.724435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.724367    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.728429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.727985    2565 generic.go:356] "Generic (PLEG): container finished" podID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerID="0e667d70396575a36ef701bb8c71517c551677509ab9716198d3f002949e2c16" exitCode=0
Sep 17 14:22:14.728429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.728115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" event={"ID":"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb","Type":"ContainerDied","Data":"0e667d70396575a36ef701bb8c71517c551677509ab9716198d3f002949e2c16"}
Sep 17 14:22:14.728788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.728547    2565 scope.go:117] "RemoveContainer" containerID="0e667d70396575a36ef701bb8c71517c551677509ab9716198d3f002949e2c16"
Sep 17 14:22:14.730700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.730271    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.730700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.730369    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.730700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.730411    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.731954 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.731621    2565 generic.go:356] "Generic (PLEG): container finished" podID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerID="1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f" exitCode=0
Sep 17 14:22:14.732264 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.732223    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" event={"ID":"6301c8c7-7838-4cc3-b149-8e1c506e3da7","Type":"ContainerDied","Data":"1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f"}
Sep 17 14:22:14.732621 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.732601    2565 scope.go:117] "RemoveContainer" containerID="1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f"
Sep 17 14:22:14.737058 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.736778    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.737058 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.736849    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.737058 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.736900    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.965361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.965319    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"]
Sep 17 14:22:14.979276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.978391    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.979276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.978475    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:14.979276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:14.978543    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:15.422632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:15.422599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:15.745304 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:15.745164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" event={"ID":"309c5898-acbe-4ad6-ba82-c3a9c51c26c3","Type":"ContainerStarted","Data":"348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553"}
Sep 17 14:22:15.751185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:15.751143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" event={"ID":"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb","Type":"ContainerStarted","Data":"0ce4d3c8f8846469189d32a8fade8e0b0073a02257be9e70622ee62fbd974bde"}
Sep 17 14:22:15.755421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:15.755383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" event={"ID":"6301c8c7-7838-4cc3-b149-8e1c506e3da7","Type":"ContainerStarted","Data":"eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d"}
Sep 17 14:22:15.758173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:15.758135    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" event={"ID":"ab092f48-a46e-498d-9f3c-369cc313e5c5","Type":"ContainerStarted","Data":"26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01"}
Sep 17 14:22:15.758322 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:15.758195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" event={"ID":"ab092f48-a46e-498d-9f3c-369cc313e5c5","Type":"ContainerStarted","Data":"ff6329bda59355793b007d4e8b079e02212fab57825b1b0007550ba1b9d6560a"}
Sep 17 14:22:15.896279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:15.890378    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" podStartSLOduration=1.890355033 podStartE2EDuration="1.890355033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:15.845350578 +0000 UTC m=+922.778855770" watchObservedRunningTime="2026-09-17 14:22:15.890355033 +0000 UTC m=+922.823860226"
Sep 17 14:22:16.255086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.254586    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:22:16.255086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.254749    2565 patch_prober.go:28] interesting pod/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 17 14:22:16.255086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.254793    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 17 14:22:16.370048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.369942    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:22:16.370715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.370677    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 17 14:22:16.370942 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.370909    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 17 14:22:16.479521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.479025    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:22:16.479521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.479174    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 17 14:22:16.479521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.479224    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 14:22:16.563694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.563600    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:16.664724 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.664684    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:16.674526 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.674493    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:16.769812 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.769784    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:16.773841 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:16.773814    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:17.255723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:17.255268    2565 patch_prober.go:28] interesting pod/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 17 14:22:17.255723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:17.255334    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 17 14:22:17.370731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:17.370233    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 17 14:22:17.370731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:17.370335    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 17 14:22:17.478146 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:17.478108    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 17 14:22:17.478464 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:17.478428    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 14:22:17.684295 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:17.684243    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:17.990128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:17.990010    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:18.232817 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:18.232765    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:18.254878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:18.254775    2565 patch_prober.go:28] interesting pod/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 17 14:22:18.255041 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:18.254878    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 17 14:22:18.398838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:18.398366    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 17 14:22:18.398838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:18.398436    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 17 14:22:18.478599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:18.478137    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 17 14:22:18.478599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:18.478209    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 14:22:19.187678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:19.187603    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:19.255242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:19.255203    2565 patch_prober.go:28] interesting pod/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 17 14:22:19.255556 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:19.255526    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 17 14:22:19.370732 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:19.370690    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 17 14:22:19.371530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:19.371052    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 17 14:22:19.478516 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:19.478415    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 17 14:22:19.478787 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:19.478754    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 14:22:20.181029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:20.178820    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:20.254939 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:20.254899    2565 patch_prober.go:28] interesting pod/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 17 14:22:20.255545 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:20.255513    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 17 14:22:20.370850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:20.370815    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 17 14:22:20.371126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:20.371092    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 17 14:22:20.478774 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:20.478662    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 17 14:22:20.478774 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:20.478737    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 14:22:21.061862 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.061817    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:21.160952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.160880    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:21.255435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.255387    2565 patch_prober.go:28] interesting pod/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 17 14:22:21.255966 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.255460    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 17 14:22:21.370946 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.370910    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 17 14:22:21.371270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.371214    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 17 14:22:21.478406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.478364    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 17 14:22:21.478711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.478678    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 14:22:21.527140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.527104    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:21.821361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.820178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:21.948853 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:21.948776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:22.254752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.254719    2565 patch_prober.go:28] interesting pod/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 17 14:22:22.255040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.254994    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 17 14:22:22.370499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.370445    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 17 14:22:22.371113 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.371065    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 17 14:22:22.479243 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.479207    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 17 14:22:22.479580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.479532    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 14:22:22.543405 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.543304    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:22.566123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.566086    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:22:22.566343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.566177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:22:22.566343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:22.566224    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:22:23.255240 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:23.255201    2565 patch_prober.go:28] interesting pod/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 17 14:22:23.255541 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:23.255509    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 17 14:22:23.370393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:23.370348    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Sep 17 14:22:23.370713 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:23.370665    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Sep 17 14:22:23.478688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:23.478649    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 17 14:22:23.478997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:23.478959    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 17 14:22:23.579066 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:23.579018    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:24.286797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:24.286764    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:22:24.294949 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:24.294919    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:22:24.379769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:24.379738    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:22:24.382028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:24.382005    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:22:24.490628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:24.490598    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:22:24.502329 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:24.502302    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:22:24.574606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:24.574570    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:24.734496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:24.734468    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:25.767957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:25.767871    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:26.561164 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:26.561124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:28.212203 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.212171    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:28.313852 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.313762    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:28.861890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.861850    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerID="a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b" exitCode=0
Sep 17 14:22:28.862105 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.861924    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" event={"ID":"7f0ec2bc-d7f5-4743-9558-065b0336f86b","Type":"ContainerDied","Data":"a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b"}
Sep 17 14:22:28.862501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.862478    2565 scope.go:117] "RemoveContainer" containerID="a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b"
Sep 17 14:22:28.863869 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.863835    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:28.863977 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.863918    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:28.863977 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.863951    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:28.864054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.863993    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d732a81-709d-46bb-9832-75dfd05875bb" containerID="871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc" exitCode=0
Sep 17 14:22:28.864054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.864032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" event={"ID":"8d732a81-709d-46bb-9832-75dfd05875bb","Type":"ContainerDied","Data":"871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc"}
Sep 17 14:22:28.864406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.864392    2565 scope.go:117] "RemoveContainer" containerID="871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc"
Sep 17 14:22:28.865449 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.865419    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:28.865551 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.865485    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:28.865551 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:28.865515    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:29.018333 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.018093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:29.118834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.118801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:29.524744 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.524708    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"]
Sep 17 14:22:29.540483 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.540451    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"
Sep 17 14:22:29.545426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.544890    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:22:29.545426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.545216    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:22:29.546074 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.546045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"]
Sep 17 14:22:29.647986 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.647933    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e58a06-a450-45d0-a460-4df11f0f300c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq\" (UID: \"b7e58a06-a450-45d0-a460-4df11f0f300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"
Sep 17 14:22:29.749605 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.749569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e58a06-a450-45d0-a460-4df11f0f300c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq\" (UID: \"b7e58a06-a450-45d0-a460-4df11f0f300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"
Sep 17 14:22:29.750717 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.750508    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e58a06-a450-45d0-a460-4df11f0f300c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq\" (UID: \"b7e58a06-a450-45d0-a460-4df11f0f300c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"
Sep 17 14:22:29.879269 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.879218    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" event={"ID":"8d732a81-709d-46bb-9832-75dfd05875bb","Type":"ContainerStarted","Data":"fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e"}
Sep 17 14:22:29.881528 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.881500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" event={"ID":"7f0ec2bc-d7f5-4743-9558-065b0336f86b","Type":"ContainerStarted","Data":"3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64"}
Sep 17 14:22:29.890896 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:29.890875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"
Sep 17 14:22:30.079107 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.079056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:30.176589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.176482    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"]
Sep 17 14:22:30.182652 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:22:30.182619    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e58a06_a450_45d0_a460_4df11f0f300c.slice/crio-33a7bffcd65c627445bbaac64c0cb652f8775358dcc123359e8100ac90d890eb WatchSource:0}: Error finding container 33a7bffcd65c627445bbaac64c0cb652f8775358dcc123359e8100ac90d890eb: Status 404 returned error can't find the container with id 33a7bffcd65c627445bbaac64c0cb652f8775358dcc123359e8100ac90d890eb
Sep 17 14:22:30.373451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.373402    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:30.375482 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.375370    2565 patch_prober.go:28] interesting pod/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 17 14:22:30.375746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.375676    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 17 14:22:30.694012 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.693318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:22:30.694676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.694649    2565 patch_prober.go:28] interesting pod/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 17 14:22:30.694858 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.694825    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 17 14:22:30.895303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:30.895235    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq" event={"ID":"b7e58a06-a450-45d0-a460-4df11f0f300c","Type":"ContainerStarted","Data":"33a7bffcd65c627445bbaac64c0cb652f8775358dcc123359e8100ac90d890eb"}
Sep 17 14:22:31.125406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:31.125376    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:31.372920 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:31.372877    2565 patch_prober.go:28] interesting pod/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 17 14:22:31.373127 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:31.372944    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 17 14:22:31.611050 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:31.611021    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:22:31.693792 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:31.693757    2565 patch_prober.go:28] interesting pod/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 17 14:22:31.694078 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:31.694046    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 17 14:22:32.089361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:32.089327    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:32.373866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:32.373770    2565 patch_prober.go:28] interesting pod/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 17 14:22:32.374152 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:32.374119    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 17 14:22:32.531176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:32.531124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:32.695167 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:32.694646    2565 patch_prober.go:28] interesting pod/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 17 14:22:32.695167 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:32.694713    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 17 14:22:33.066336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:33.054680    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:33.373277 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:33.372904    2565 patch_prober.go:28] interesting pod/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 17 14:22:33.373277 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:33.372970    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 17 14:22:33.630524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:33.630427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:33.696275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:33.693281    2565 patch_prober.go:28] interesting pod/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 17 14:22:33.696275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:33.693340    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 17 14:22:33.880969 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:33.880781    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:34.375662 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:34.375623    2565 patch_prober.go:28] interesting pod/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 17 14:22:34.376363 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:34.375943    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 17 14:22:34.692913 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:34.692814    2565 patch_prober.go:28] interesting pod/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Sep 17 14:22:34.692913 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:34.692887    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Sep 17 14:22:34.816160 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:34.816122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:35.376073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:35.375635    2565 patch_prober.go:28] interesting pod/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body=
Sep 17 14:22:35.376073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:35.375699    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused"
Sep 17 14:22:35.663147 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:35.663014    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:35.720273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:35.720218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:22:35.722927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:35.722855    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:22:36.098244 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:36.098172    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:36.191124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:36.191093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:36.378216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:36.376955    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:36.382690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:36.382664    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:36.384173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:36.384152    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:36.959313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:36.959220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq" event={"ID":"b7e58a06-a450-45d0-a460-4df11f0f300c","Type":"ContainerStarted","Data":"966eec57251d5cd933e2ec0c03bce2f8005c3d19757589c6dbd414b027739ed1"}
Sep 17 14:22:37.970009 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:37.969967    2565 generic.go:356] "Generic (PLEG): container finished" podID="b7e58a06-a450-45d0-a460-4df11f0f300c" containerID="966eec57251d5cd933e2ec0c03bce2f8005c3d19757589c6dbd414b027739ed1" exitCode=0
Sep 17 14:22:37.970532 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:37.970120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq" event={"ID":"b7e58a06-a450-45d0-a460-4df11f0f300c","Type":"ContainerDied","Data":"966eec57251d5cd933e2ec0c03bce2f8005c3d19757589c6dbd414b027739ed1"}
Sep 17 14:22:39.127471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:39.127441    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"
Sep 17 14:22:39.224865 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:39.224827    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e58a06-a450-45d0-a460-4df11f0f300c-maas-api-service-ca\") pod \"b7e58a06-a450-45d0-a460-4df11f0f300c\" (UID: \"b7e58a06-a450-45d0-a460-4df11f0f300c\") "
Sep 17 14:22:39.225289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:39.225244    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e58a06-a450-45d0-a460-4df11f0f300c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7e58a06-a450-45d0-a460-4df11f0f300c" (UID: "b7e58a06-a450-45d0-a460-4df11f0f300c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:22:39.326878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:39.326846    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e58a06-a450-45d0-a460-4df11f0f300c-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:39.984120 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:39.984093    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"
Sep 17 14:22:39.984285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:39.984089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq" event={"ID":"b7e58a06-a450-45d0-a460-4df11f0f300c","Type":"ContainerDied","Data":"33a7bffcd65c627445bbaac64c0cb652f8775358dcc123359e8100ac90d890eb"}
Sep 17 14:22:39.984285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:39.984224    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a7bffcd65c627445bbaac64c0cb652f8775358dcc123359e8100ac90d890eb"
Sep 17 14:22:41.997595 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:41.997554    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerID="26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01" exitCode=0
Sep 17 14:22:41.998053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:41.997610    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" event={"ID":"ab092f48-a46e-498d-9f3c-369cc313e5c5","Type":"ContainerDied","Data":"26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01"}
Sep 17 14:22:41.998200 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:41.998179    2565 scope.go:117] "RemoveContainer" containerID="26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01"
Sep 17 14:22:41.999854 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:41.999820    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:41.999931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:41.999916    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:41.999985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:41.999968    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:42.467765 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:42.467722    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:43.005655 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:43.005621    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" event={"ID":"ab092f48-a46e-498d-9f3c-369cc313e5c5","Type":"ContainerStarted","Data":"998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17"}
Sep 17 14:22:43.662804 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:43.662772    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:43.663935 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:43.663904    2565 patch_prober.go:28] interesting pod/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 17 14:22:43.664170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:43.664140    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 14:22:43.691492 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:43.691462    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:43.783112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:43.783086    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:43.883583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:43.883545    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:44.663597 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:44.663563    2565 patch_prober.go:28] interesting pod/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 17 14:22:44.664168 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:44.664136    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 14:22:45.662868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:45.662824    2565 patch_prober.go:28] interesting pod/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 17 14:22:45.663079 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:45.662894    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 14:22:45.694107 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:45.694076    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:46.428984 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.428950    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:46.487309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.487272    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"]
Sep 17 14:22:46.487862 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.487826    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9" podUID="0419a270-844e-4a9e-8c44-b4abb8d160dd" containerName="payload-processing" containerID="cri-o://10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce" gracePeriod=30
Sep 17 14:22:46.579448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.576740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:46.683366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.676152    2565 patch_prober.go:28] interesting pod/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 17 14:22:46.683366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.676207    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 14:22:46.683366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.677683    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:46.711692 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.711661    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"]
Sep 17 14:22:46.715661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.715609    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a48a4ae7-slkxq"]
Sep 17 14:22:46.771459 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.771422    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:46.813212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.812952    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:46.842938 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.842908    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"]
Sep 17 14:22:46.843446 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.843405    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy" containerID="cri-o://fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e" gracePeriod=30
Sep 17 14:22:46.871554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.871521    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:46.916700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.914796    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0419a270-844e-4a9e-8c44-b4abb8d160dd-plugins-config-volume\") pod \"0419a270-844e-4a9e-8c44-b4abb8d160dd\" (UID: \"0419a270-844e-4a9e-8c44-b4abb8d160dd\") "
Sep 17 14:22:46.916700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.915368    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0419a270-844e-4a9e-8c44-b4abb8d160dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0419a270-844e-4a9e-8c44-b4abb8d160dd" (UID: "0419a270-844e-4a9e-8c44-b4abb8d160dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:22:46.916700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.915626    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tf9tt\" (UniqueName: \"kubernetes.io/projected/0419a270-844e-4a9e-8c44-b4abb8d160dd-kube-api-access-tf9tt\") pod \"0419a270-844e-4a9e-8c44-b4abb8d160dd\" (UID: \"0419a270-844e-4a9e-8c44-b4abb8d160dd\") "
Sep 17 14:22:46.916700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.916451    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0419a270-844e-4a9e-8c44-b4abb8d160dd-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:46.917965 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.917939    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0419a270-844e-4a9e-8c44-b4abb8d160dd-kube-api-access-tf9tt" (OuterVolumeSpecName: "kube-api-access-tf9tt") pod "0419a270-844e-4a9e-8c44-b4abb8d160dd" (UID: "0419a270-844e-4a9e-8c44-b4abb8d160dd"). InnerVolumeSpecName "kube-api-access-tf9tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:22:46.964610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:46.964474    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:47.017506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.017471    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tf9tt\" (UniqueName: \"kubernetes.io/projected/0419a270-844e-4a9e-8c44-b4abb8d160dd-kube-api-access-tf9tt\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:47.031391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.031354    2565 generic.go:356] "Generic (PLEG): container finished" podID="0419a270-844e-4a9e-8c44-b4abb8d160dd" containerID="10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce" exitCode=0
Sep 17 14:22:47.031660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.031463    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"
Sep 17 14:22:47.032334 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.031485    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9" event={"ID":"0419a270-844e-4a9e-8c44-b4abb8d160dd","Type":"ContainerDied","Data":"10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce"}
Sep 17 14:22:47.032521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.032501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9" event={"ID":"0419a270-844e-4a9e-8c44-b4abb8d160dd","Type":"ContainerDied","Data":"560bfed2a0376204da9692caa4447af61af670ffc88718e67aa71077ef421d8e"}
Sep 17 14:22:47.032645 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.032555    2565 scope.go:117] "RemoveContainer" containerID="10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce"
Sep 17 14:22:47.044017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.044000    2565 scope.go:117] "RemoveContainer" containerID="10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce"
Sep 17 14:22:47.058275 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:47.056358    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce\": container with ID starting with 10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce not found: ID does not exist" containerID="10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce"
Sep 17 14:22:47.058275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.056400    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce"} err="failed to get container status \"10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce\": rpc error: code = NotFound desc = could not find container \"10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce\": container with ID starting with 10d1ec8bb6a268453510f4efed0630cf7f9c838016856d3dbc89804aca4fa7ce not found: ID does not exist"
Sep 17 14:22:47.080277 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.079605    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"]
Sep 17 14:22:47.087546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.087513    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a48a4ae7-7569987ddf-7gbn9"]
Sep 17 14:22:47.426410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.426367    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:47.520401 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.520366    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0419a270-844e-4a9e-8c44-b4abb8d160dd" path="/var/lib/kubelet/pods/0419a270-844e-4a9e-8c44-b4abb8d160dd/volumes"
Sep 17 14:22:47.520773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.520758    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e58a06-a450-45d0-a460-4df11f0f300c" path="/var/lib/kubelet/pods/b7e58a06-a450-45d0-a460-4df11f0f300c/volumes"
Sep 17 14:22:47.577370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.577317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:47.663542 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.663507    2565 patch_prober.go:28] interesting pod/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 17 14:22:47.663801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.663772    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 17 14:22:47.818070 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.817977    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:47.910341 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.910305    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"]
Sep 17 14:22:47.910977 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.910942    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0419a270-844e-4a9e-8c44-b4abb8d160dd" containerName="payload-processing"
Sep 17 14:22:47.911168 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.911154    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0419a270-844e-4a9e-8c44-b4abb8d160dd" containerName="payload-processing"
Sep 17 14:22:47.911320 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.911307    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e58a06-a450-45d0-a460-4df11f0f300c" containerName="revoke-keys"
Sep 17 14:22:47.911437 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.911425    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e58a06-a450-45d0-a460-4df11f0f300c" containerName="revoke-keys"
Sep 17 14:22:47.911633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.911616    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0419a270-844e-4a9e-8c44-b4abb8d160dd" containerName="payload-processing"
Sep 17 14:22:47.911717 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.911638    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e58a06-a450-45d0-a460-4df11f0f300c" containerName="revoke-keys"
Sep 17 14:22:47.914901 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.914877    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:47.917965 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.917943    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-7f7de479-gw-openshift-default-dockercfg-2lpww\""
Sep 17 14:22:47.933284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.933247    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"]
Sep 17 14:22:47.976594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:47.976553    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028068    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-socket\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028116    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-podinfo\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-data\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-token\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028395    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-envoy\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-certs\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgt5p\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-kube-api-access-jgt5p\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-credential-socket\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.030468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.028669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420a4aa0-df86-4b75-bf0e-b73ac1956693-istiod-ca-cert\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-socket\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-podinfo\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130179    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-data\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-token\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-envoy\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-certs\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgt5p\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-kube-api-access-jgt5p\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-credential-socket\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420a4aa0-df86-4b75-bf0e-b73ac1956693-istiod-ca-cert\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130751 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-data\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.130847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.130780    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-socket\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.131078 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.131048    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-certs\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.131575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.131550    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420a4aa0-df86-4b75-bf0e-b73ac1956693-istiod-ca-cert\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.131575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.131556    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-credential-socket\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.132762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.132739    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-envoy\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.132945 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.132881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-podinfo\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.141990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.141970    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-token\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.142246 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.142224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgt5p\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-kube-api-access-jgt5p\") pod \"e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") " pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.227924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.227885    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:48.292859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.292788    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:48.550690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.550662    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"]
Sep 17 14:22:48.553558 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:22:48.553521    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod420a4aa0_df86_4b75_bf0e_b73ac1956693.slice/crio-d91341200d2f214eecf37d7ec3b9617e4afb57203ff2b26e7bb1120ce8ad6840 WatchSource:0}: Error finding container d91341200d2f214eecf37d7ec3b9617e4afb57203ff2b26e7bb1120ce8ad6840: Status 404 returned error can't find the container with id d91341200d2f214eecf37d7ec3b9617e4afb57203ff2b26e7bb1120ce8ad6840
Sep 17 14:22:48.556391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.556357    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:48.556500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.556433    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:48.556500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.556479    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:22:48.667407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.667377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:48.668610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.668578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:22:48.707601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:48.707568    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:49.047951 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:49.047916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" event={"ID":"420a4aa0-df86-4b75-bf0e-b73ac1956693","Type":"ContainerStarted","Data":"269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f"}
Sep 17 14:22:49.048450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:49.047981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" event={"ID":"420a4aa0-df86-4b75-bf0e-b73ac1956693","Type":"ContainerStarted","Data":"d91341200d2f214eecf37d7ec3b9617e4afb57203ff2b26e7bb1120ce8ad6840"}
Sep 17 14:22:49.228519 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:49.228484    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:49.234564 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:49.234538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:49.289495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:49.289438    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" podStartSLOduration=2.289420651 podStartE2EDuration="2.289420651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:49.094347342 +0000 UTC m=+956.027852533" watchObservedRunningTime="2026-09-17 14:22:49.289420651 +0000 UTC m=+956.222925844"
Sep 17 14:22:50.054435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:50.054404    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:50.055548 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:50.055524    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:22:50.381560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:50.381528    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:50.791647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:50.791569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:50.936291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:50.936245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:51.385216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:51.385179    2565 patch_prober.go:28] interesting pod/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:22:51.385606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:51.385234    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:22:51.404887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:51.404858    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:52.310936 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.310910    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:52.376548 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-certs\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.376691 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376618    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d732a81-709d-46bb-9832-75dfd05875bb-istiod-ca-cert\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.376691 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-envoy\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.376810 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376691    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crxfm\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-kube-api-access-crxfm\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.376810 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-data\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.376810 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376751    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-socket\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.376810 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376717    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:52.377017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376818    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d732a81-709d-46bb-9832-75dfd05875bb-istio-podinfo\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.377017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376854    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-istio-token\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.377017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376928    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-credential-socket\") pod \"8d732a81-709d-46bb-9832-75dfd05875bb\" (UID: \"8d732a81-709d-46bb-9832-75dfd05875bb\") "
Sep 17 14:22:52.377017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.376948    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:52.377017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.377000    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d732a81-709d-46bb-9832-75dfd05875bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:22:52.377362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.377009    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:52.377362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.377200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:52.377447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.377372    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d732a81-709d-46bb-9832-75dfd05875bb-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.377447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.377397    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.377447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.377415    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.377447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.377431    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.377447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.377445    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.378932 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.378906    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d732a81-709d-46bb-9832-75dfd05875bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:22:52.379331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.379302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:22:52.379558 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.379535    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-kube-api-access-crxfm" (OuterVolumeSpecName: "kube-api-access-crxfm") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "kube-api-access-crxfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:22:52.379641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.379567    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d732a81-709d-46bb-9832-75dfd05875bb" (UID: "8d732a81-709d-46bb-9832-75dfd05875bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:22:52.422531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.422507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:52.478887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.478855    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d732a81-709d-46bb-9832-75dfd05875bb-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.478887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.478882    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crxfm\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-kube-api-access-crxfm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.478887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.478894    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d732a81-709d-46bb-9832-75dfd05875bb-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.479120 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.478903    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d732a81-709d-46bb-9832-75dfd05875bb-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:22:52.620450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.620418    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:52.731188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:52.731116    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:53.072468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.072380    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d732a81-709d-46bb-9832-75dfd05875bb" containerID="fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e" exitCode=0
Sep 17 14:22:53.072468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.072429    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"
Sep 17 14:22:53.072468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.072460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" event={"ID":"8d732a81-709d-46bb-9832-75dfd05875bb","Type":"ContainerDied","Data":"fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e"}
Sep 17 14:22:53.072742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.072509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7" event={"ID":"8d732a81-709d-46bb-9832-75dfd05875bb","Type":"ContainerDied","Data":"c0f85dc14d0e62e0e8f4dca67e0715dbc6a2d52e40657f40ad5929e9d1d9b94f"}
Sep 17 14:22:53.072742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.072533    2565 scope.go:117] "RemoveContainer" containerID="fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e"
Sep 17 14:22:53.082524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.082507    2565 scope.go:117] "RemoveContainer" containerID="871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc"
Sep 17 14:22:53.090492 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.090471    2565 scope.go:117] "RemoveContainer" containerID="fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e"
Sep 17 14:22:53.090739 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:53.090719    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e\": container with ID starting with fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e not found: ID does not exist" containerID="fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e"
Sep 17 14:22:53.090789 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.090747    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e"} err="failed to get container status \"fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e\": rpc error: code = NotFound desc = could not find container \"fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e\": container with ID starting with fe815ffcfdbcdac64a00988a3f851beb66967977c15ff174b698b8fc2cfc562e not found: ID does not exist"
Sep 17 14:22:53.090789 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.090764    2565 scope.go:117] "RemoveContainer" containerID="871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc"
Sep 17 14:22:53.091044 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:53.091028    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc\": container with ID starting with 871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc not found: ID does not exist" containerID="871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc"
Sep 17 14:22:53.091106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.091046    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc"} err="failed to get container status \"871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc\": rpc error: code = NotFound desc = could not find container \"871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc\": container with ID starting with 871e8b260f29291398f3bfe262887ae8cb2320aaeaf1fe3efeb1a88c86b868fc not found: ID does not exist"
Sep 17 14:22:53.097111 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.097091    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"]
Sep 17 14:22:53.101974 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.101952    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a48a4ae7-openshift-default-778678f8db-2hgh7"]
Sep 17 14:22:53.306791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.306757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:53.520690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:53.520652    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" path="/var/lib/kubelet/pods/8d732a81-709d-46bb-9832-75dfd05875bb/volumes"
Sep 17 14:22:54.350778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:54.350724    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:54.458612 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:54.458577    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:55.279524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.279480    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:55.414070 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.414042    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:55.530668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.530592    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"]
Sep 17 14:22:55.531006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.530994    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy"
Sep 17 14:22:55.531052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.531008    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy"
Sep 17 14:22:55.531052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.531024    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy"
Sep 17 14:22:55.531052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.531030    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy"
Sep 17 14:22:55.531151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.531091    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy"
Sep 17 14:22:55.531151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.531100    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d732a81-709d-46bb-9832-75dfd05875bb" containerName="istio-proxy"
Sep 17 14:22:55.534497 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.534476    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:55.536869 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:55.536832    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" podUID="c1137789-bce2-4cbb-84a2-7747dfe73e29" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:55.536977 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:55.536873    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"payload-processing-plugins-e2e-migrate-7f7de479\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7f7de479\"" type="*v1.ConfigMap"
Sep 17 14:22:55.537895 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:55.537871    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"payload-processing-e2e-migrate-7f7de479-dockercfg-mt9vx\" is forbidden: User \"system:node:ip-10-0-128-175.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-175.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7f7de479-dockercfg-mt9vx\"" type="*v1.Secret"
Sep 17 14:22:55.551668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.551645    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"]
Sep 17 14:22:55.613775 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.613706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7t7\" (UniqueName: \"kubernetes.io/projected/c1137789-bce2-4cbb-84a2-7747dfe73e29-kube-api-access-tq7t7\") pod \"payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:55.614416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.614384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:55.624737 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.624706    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"]
Sep 17 14:22:55.628655 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.628624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:55.647665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.647640    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"]
Sep 17 14:22:55.718940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.718901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvm9b\" (UniqueName: \"kubernetes.io/projected/29a9225f-f239-471d-a7be-19228d814c80-kube-api-access-gvm9b\") pod \"payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:55.719110 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.719081    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7t7\" (UniqueName: \"kubernetes.io/projected/c1137789-bce2-4cbb-84a2-7747dfe73e29-kube-api-access-tq7t7\") pod \"payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:55.719177 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.719138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:55.719231 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.719192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:55.741978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.741951    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7t7\" (UniqueName: \"kubernetes.io/projected/c1137789-bce2-4cbb-84a2-7747dfe73e29-kube-api-access-tq7t7\") pod \"payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:55.820490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.820403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:55.820656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.820499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvm9b\" (UniqueName: \"kubernetes.io/projected/29a9225f-f239-471d-a7be-19228d814c80-kube-api-access-gvm9b\") pod \"payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:55.829244 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:55.829218    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvm9b\" (UniqueName: \"kubernetes.io/projected/29a9225f-f239-471d-a7be-19228d814c80-kube-api-access-gvm9b\") pod \"payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:56.092277 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:56.092210    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:56.720411 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:56.720365    2565 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-migrate-7f7de479: failed to sync configmap cache: timed out waiting for the condition
Sep 17 14:22:56.720872 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:56.720458    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume podName:c1137789-bce2-4cbb-84a2-7747dfe73e29 nodeName:}" failed. No retries permitted until 2026-09-17 14:22:57.220438038 +0000 UTC m=+964.153943207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume") pod "payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh" (UID: "c1137789-bce2-4cbb-84a2-7747dfe73e29") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 14:22:56.821627 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:56.821598    2565 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-migrate-7f7de479: failed to sync configmap cache: timed out waiting for the condition
Sep 17 14:22:56.821765 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:22:56.821670    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume podName:29a9225f-f239-471d-a7be-19228d814c80 nodeName:}" failed. No retries permitted until 2026-09-17 14:22:57.321654795 +0000 UTC m=+964.255159965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume") pod "payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr" (UID: "29a9225f-f239-471d-a7be-19228d814c80") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 14:22:56.938325 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:56.938294    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:57.004577 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.004502    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-7f7de479-dockercfg-mt9vx\""
Sep 17 14:22:57.055369 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.055343    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-7f7de479\""
Sep 17 14:22:57.234890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.234858    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:57.235546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.235528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:57.336563 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.336523    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:57.337323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.337297    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume\") pod \"payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") " pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:57.344297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.344275    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:57.440746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.440713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:57.626707 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.626677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"]
Sep 17 14:22:57.627102 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:22:57.627078    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1137789_bce2_4cbb_84a2_7747dfe73e29.slice/crio-1a1d26ce2993f8fff0a86e2b960d37fd28cfd1b7ef9106ef3f4ec57a35b94584 WatchSource:0}: Error finding container 1a1d26ce2993f8fff0a86e2b960d37fd28cfd1b7ef9106ef3f4ec57a35b94584: Status 404 returned error can't find the container with id 1a1d26ce2993f8fff0a86e2b960d37fd28cfd1b7ef9106ef3f4ec57a35b94584
Sep 17 14:22:57.752399 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.752368    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"]
Sep 17 14:22:57.753091 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:22:57.753052    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a9225f_f239_471d_a7be_19228d814c80.slice/crio-39a9983d121a9b944ce198db0bd7bd24873c6164d54c9df711f3a356a606edc0 WatchSource:0}: Error finding container 39a9983d121a9b944ce198db0bd7bd24873c6164d54c9df711f3a356a606edc0: Status 404 returned error can't find the container with id 39a9983d121a9b944ce198db0bd7bd24873c6164d54c9df711f3a356a606edc0
Sep 17 14:22:57.929063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:57.928970    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:22:58.106722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:58.106495    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr" event={"ID":"29a9225f-f239-471d-a7be-19228d814c80","Type":"ContainerStarted","Data":"f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57"}
Sep 17 14:22:58.106722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:58.106569    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:22:58.106722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:58.106597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr" event={"ID":"29a9225f-f239-471d-a7be-19228d814c80","Type":"ContainerStarted","Data":"39a9983d121a9b944ce198db0bd7bd24873c6164d54c9df711f3a356a606edc0"}
Sep 17 14:22:58.108501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:58.108470    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh" event={"ID":"c1137789-bce2-4cbb-84a2-7747dfe73e29","Type":"ContainerStarted","Data":"32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced"}
Sep 17 14:22:58.108635 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:58.108530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh" event={"ID":"c1137789-bce2-4cbb-84a2-7747dfe73e29","Type":"ContainerStarted","Data":"1a1d26ce2993f8fff0a86e2b960d37fd28cfd1b7ef9106ef3f4ec57a35b94584"}
Sep 17 14:22:58.108635 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:58.108553    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:22:58.131920 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:58.131874    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr" podStartSLOduration=3.131859638 podStartE2EDuration="3.131859638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:58.130027465 +0000 UTC m=+965.063532656" watchObservedRunningTime="2026-09-17 14:22:58.131859638 +0000 UTC m=+965.065364827"
Sep 17 14:22:58.152393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:22:58.152289    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh" podStartSLOduration=3.152242838 podStartE2EDuration="3.152242838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:22:58.150303989 +0000 UTC m=+965.083809179" watchObservedRunningTime="2026-09-17 14:22:58.152242838 +0000 UTC m=+965.085748030"
Sep 17 14:23:09.116193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:09.116156    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:23:09.116620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:09.116239    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:23:14.524391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.524304    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"]
Sep 17 14:23:14.526909 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.526891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.531062 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.531042    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"kube-root-ca.crt\""
Sep 17 14:23:14.531728 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.531704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"default-dockercfg-pcxz4\""
Sep 17 14:23:14.531858 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.531813    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"simulator-w4-8b50f2-kserve-self-signed-certs\""
Sep 17 14:23:14.531920 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.531849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"openshift-service-ca.crt\""
Sep 17 14:23:14.547425 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.547390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"]
Sep 17 14:23:14.561313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.561282    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:14.632439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.632406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tmp-dir\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.632604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.632462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-dshm\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.632604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.632511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrhf\" (UniqueName: \"kubernetes.io/projected/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-kube-api-access-mzrhf\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.632604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.632595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-model-cache\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.632723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.632618    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tls-certs\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.632723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.632634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-home\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.733958 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.733924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-model-cache\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.733958 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.733959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tls-certs\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.734173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.733978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-home\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.734173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.734043    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tmp-dir\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.734173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.734084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-dshm\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.734173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.734128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrhf\" (UniqueName: \"kubernetes.io/projected/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-kube-api-access-mzrhf\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.734430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.734345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-model-cache\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.734430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.734411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-home\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.734515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.734447    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tmp-dir\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.736303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.736280    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-dshm\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.736509 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.736491    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tls-certs\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.749683 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.749660    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrhf\" (UniqueName: \"kubernetes.io/projected/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-kube-api-access-mzrhf\") pod \"simulator-w4-8b50f2-kserve-875d497b4-t94rr\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") " pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:14.836889 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:14.836857    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:15.130228 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.130139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"]
Sep 17 14:23:15.136744 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:23:15.136712    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd59b729d_1f73_4ac4_a9a8_9bc05f32e70b.slice/crio-6425f7adc075b39bc6842605920f685287be4c3b2e2f916b596d4c9ff84248c7 WatchSource:0}: Error finding container 6425f7adc075b39bc6842605920f685287be4c3b2e2f916b596d4c9ff84248c7: Status 404 returned error can't find the container with id 6425f7adc075b39bc6842605920f685287be4c3b2e2f916b596d4c9ff84248c7
Sep 17 14:23:15.232485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.232457    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr" event={"ID":"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b","Type":"ContainerStarted","Data":"7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c"}
Sep 17 14:23:15.232620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.232500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr" event={"ID":"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b","Type":"ContainerStarted","Data":"6425f7adc075b39bc6842605920f685287be4c3b2e2f916b596d4c9ff84248c7"}
Sep 17 14:23:15.234324 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.234293    2565 generic.go:356] "Generic (PLEG): container finished" podID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerID="269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f" exitCode=0
Sep 17 14:23:15.234458 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.234355    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" event={"ID":"420a4aa0-df86-4b75-bf0e-b73ac1956693","Type":"ContainerDied","Data":"269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f"}
Sep 17 14:23:15.234828 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.234810    2565 scope.go:117] "RemoveContainer" containerID="269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f"
Sep 17 14:23:15.235974 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.235941    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:23:15.236055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.236012    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:23:15.236055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.236042    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:23:15.284970 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:15.284922    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr" podStartSLOduration=1.284901493 podStartE2EDuration="1.284901493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:23:15.281745804 +0000 UTC m=+982.215251010" watchObservedRunningTime="2026-09-17 14:23:15.284901493 +0000 UTC m=+982.218406685"
Sep 17 14:23:16.244962 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:16.244867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" event={"ID":"420a4aa0-df86-4b75-bf0e-b73ac1956693","Type":"ContainerStarted","Data":"80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b"}
Sep 17 14:23:17.228888 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:17.228849    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:23:17.230099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:17.230071    2565 patch_prober.go:28] interesting pod/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 17 14:23:17.230245 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:17.230128    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 17 14:23:17.567472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:17.567344    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:18.228594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:18.228554    2565 patch_prober.go:28] interesting pod/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 17 14:23:18.228788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:18.228624    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 17 14:23:18.463523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:18.463487    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:19.228883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:19.228848    2565 patch_prober.go:28] interesting pod/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 17 14:23:19.229295 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:19.228915    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 17 14:23:19.279590 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:19.279554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:20.055407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.055370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:23:20.228415 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.228383    2565 patch_prober.go:28] interesting pod/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 17 14:23:20.228588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.228432    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 17 14:23:20.278749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.278683    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:20.827795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.827762    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"]
Sep 17 14:23:20.830354 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.830328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"
Sep 17 14:23:20.833904 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.833882    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:23:20.834045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.833905    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:23:20.850106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.850083    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"]
Sep 17 14:23:20.905396 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:20.905369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9996f7-e318-48ed-b23c-e31e8a11239f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd\" (UID: \"0a9996f7-e318-48ed-b23c-e31e8a11239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"
Sep 17 14:23:21.006485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:21.006449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9996f7-e318-48ed-b23c-e31e8a11239f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd\" (UID: \"0a9996f7-e318-48ed-b23c-e31e8a11239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"
Sep 17 14:23:21.007050 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:21.007029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9996f7-e318-48ed-b23c-e31e8a11239f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd\" (UID: \"0a9996f7-e318-48ed-b23c-e31e8a11239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"
Sep 17 14:23:21.141460 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:21.141358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"
Sep 17 14:23:21.229241 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:21.229208    2565 patch_prober.go:28] interesting pod/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 17 14:23:21.229386 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:21.229279    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 17 14:23:21.355861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:21.355823    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:21.423356 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:21.423323    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"]
Sep 17 14:23:21.425015 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:23:21.424980    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9996f7_e318_48ed_b23c_e31e8a11239f.slice/crio-2ba763535fe8f73a86e8b5eaec3766399e89102c99e94c6548c9cce529fcb17c WatchSource:0}: Error finding container 2ba763535fe8f73a86e8b5eaec3766399e89102c99e94c6548c9cce529fcb17c: Status 404 returned error can't find the container with id 2ba763535fe8f73a86e8b5eaec3766399e89102c99e94c6548c9cce529fcb17c
Sep 17 14:23:22.233224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:22.233193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:23:22.234202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:22.234183    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:23:22.292246 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:22.292212    2565 generic.go:356] "Generic (PLEG): container finished" podID="0a9996f7-e318-48ed-b23c-e31e8a11239f" containerID="d2889e117e1e4a4432385780aff3d5c89a77c011fc8d9ffbb995c0990d46fc69" exitCode=0
Sep 17 14:23:22.292367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:22.292279    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd" event={"ID":"0a9996f7-e318-48ed-b23c-e31e8a11239f","Type":"ContainerDied","Data":"d2889e117e1e4a4432385780aff3d5c89a77c011fc8d9ffbb995c0990d46fc69"}
Sep 17 14:23:22.292367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:22.292318    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd" event={"ID":"0a9996f7-e318-48ed-b23c-e31e8a11239f","Type":"ContainerStarted","Data":"2ba763535fe8f73a86e8b5eaec3766399e89102c99e94c6548c9cce529fcb17c"}
Sep 17 14:23:22.333434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:22.333406    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:23.449262 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:23.449222    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"
Sep 17 14:23:23.531676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:23.531643    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9996f7-e318-48ed-b23c-e31e8a11239f-maas-api-service-ca\") pod \"0a9996f7-e318-48ed-b23c-e31e8a11239f\" (UID: \"0a9996f7-e318-48ed-b23c-e31e8a11239f\") "
Sep 17 14:23:23.531966 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:23.531944    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9996f7-e318-48ed-b23c-e31e8a11239f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0a9996f7-e318-48ed-b23c-e31e8a11239f" (UID: "0a9996f7-e318-48ed-b23c-e31e8a11239f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:23:23.569395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:23.569311    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:23.632823 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:23.632789    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9996f7-e318-48ed-b23c-e31e8a11239f-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:24.306604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:24.306575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"
Sep 17 14:23:24.306813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:24.306611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd" event={"ID":"0a9996f7-e318-48ed-b23c-e31e8a11239f","Type":"ContainerDied","Data":"2ba763535fe8f73a86e8b5eaec3766399e89102c99e94c6548c9cce529fcb17c"}
Sep 17 14:23:24.306813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:24.306663    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ba763535fe8f73a86e8b5eaec3766399e89102c99e94c6548c9cce529fcb17c"
Sep 17 14:23:24.567671 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:24.567590    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:24.837735 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:24.837705    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:24.837905 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:24.837754    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:24.851121 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:24.851097    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:25.324012 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:25.323982    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:23:25.478458 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:25.478424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:26.285696 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:26.285666    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:27.094391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:27.094356    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:31.989699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:31.989667    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:32.923855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:32.923822    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:32.955026 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:32.953736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"]
Sep 17 14:23:32.955026 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:32.954023    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr" podUID="29a9225f-f239-471d-a7be-19228d814c80" containerName="payload-processing" containerID="cri-o://f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57" gracePeriod=30
Sep 17 14:23:32.964770 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:32.964736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"]
Sep 17 14:23:32.965001 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:32.964975    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh" podUID="c1137789-bce2-4cbb-84a2-7747dfe73e29" containerName="payload-pre-processing" containerID="cri-o://32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced" gracePeriod=30
Sep 17 14:23:33.028761 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.028724    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:33.144511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.144482    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:33.193681 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.193239    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"]
Sep 17 14:23:33.200914 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.200888    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-7f7de479-gvbdd"]
Sep 17 14:23:33.218701 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.218683    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:23:33.235785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.235754    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:33.243392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.243374    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:23:33.339699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.339282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvm9b\" (UniqueName: \"kubernetes.io/projected/29a9225f-f239-471d-a7be-19228d814c80-kube-api-access-gvm9b\") pod \"29a9225f-f239-471d-a7be-19228d814c80\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") "
Sep 17 14:23:33.339699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.339343    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq7t7\" (UniqueName: \"kubernetes.io/projected/c1137789-bce2-4cbb-84a2-7747dfe73e29-kube-api-access-tq7t7\") pod \"c1137789-bce2-4cbb-84a2-7747dfe73e29\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") "
Sep 17 14:23:33.340040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.340018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume\") pod \"c1137789-bce2-4cbb-84a2-7747dfe73e29\" (UID: \"c1137789-bce2-4cbb-84a2-7747dfe73e29\") "
Sep 17 14:23:33.340921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.340901    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume\") pod \"29a9225f-f239-471d-a7be-19228d814c80\" (UID: \"29a9225f-f239-471d-a7be-19228d814c80\") "
Sep 17 14:23:33.341732 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.340779    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c1137789-bce2-4cbb-84a2-7747dfe73e29" (UID: "c1137789-bce2-4cbb-84a2-7747dfe73e29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:23:33.341732 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.341519    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1137789-bce2-4cbb-84a2-7747dfe73e29-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:33.341732 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.341688    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "29a9225f-f239-471d-a7be-19228d814c80" (UID: "29a9225f-f239-471d-a7be-19228d814c80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:23:33.345143 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.345105    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1137789-bce2-4cbb-84a2-7747dfe73e29-kube-api-access-tq7t7" (OuterVolumeSpecName: "kube-api-access-tq7t7") pod "c1137789-bce2-4cbb-84a2-7747dfe73e29" (UID: "c1137789-bce2-4cbb-84a2-7747dfe73e29"). InnerVolumeSpecName "kube-api-access-tq7t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:23:33.346312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.346285    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a9225f-f239-471d-a7be-19228d814c80-kube-api-access-gvm9b" (OuterVolumeSpecName: "kube-api-access-gvm9b") pod "29a9225f-f239-471d-a7be-19228d814c80" (UID: "29a9225f-f239-471d-a7be-19228d814c80"). InnerVolumeSpecName "kube-api-access-gvm9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:23:33.362508 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.362435    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"]
Sep 17 14:23:33.362760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.362707    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy" containerID="cri-o://80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b" gracePeriod=30
Sep 17 14:23:33.369216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.368896    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"
Sep 17 14:23:33.369216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.368911    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh" event={"ID":"c1137789-bce2-4cbb-84a2-7747dfe73e29","Type":"ContainerDied","Data":"32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced"}
Sep 17 14:23:33.369216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.368985    2565 scope.go:117] "RemoveContainer" containerID="32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced"
Sep 17 14:23:33.369634 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.368802    2565 generic.go:356] "Generic (PLEG): container finished" podID="c1137789-bce2-4cbb-84a2-7747dfe73e29" containerID="32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced" exitCode=0
Sep 17 14:23:33.369911 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.369830    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh" event={"ID":"c1137789-bce2-4cbb-84a2-7747dfe73e29","Type":"ContainerDied","Data":"1a1d26ce2993f8fff0a86e2b960d37fd28cfd1b7ef9106ef3f4ec57a35b94584"}
Sep 17 14:23:33.373656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.373618    2565 generic.go:356] "Generic (PLEG): container finished" podID="29a9225f-f239-471d-a7be-19228d814c80" containerID="f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57" exitCode=0
Sep 17 14:23:33.373762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.373671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr" event={"ID":"29a9225f-f239-471d-a7be-19228d814c80","Type":"ContainerDied","Data":"f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57"}
Sep 17 14:23:33.373762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.373708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr" event={"ID":"29a9225f-f239-471d-a7be-19228d814c80","Type":"ContainerDied","Data":"39a9983d121a9b944ce198db0bd7bd24873c6164d54c9df711f3a356a606edc0"}
Sep 17 14:23:33.373880 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.373787    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"
Sep 17 14:23:33.383578 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.383558    2565 scope.go:117] "RemoveContainer" containerID="32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced"
Sep 17 14:23:33.383974 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:23:33.383912    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced\": container with ID starting with 32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced not found: ID does not exist" containerID="32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced"
Sep 17 14:23:33.383974 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.383951    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced"} err="failed to get container status \"32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced\": rpc error: code = NotFound desc = could not find container \"32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced\": container with ID starting with 32b1d0eb0b987eb6cd4182cbfaab58bebd2d09a37b16225583fa7484268ddced not found: ID does not exist"
Sep 17 14:23:33.384124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.383975    2565 scope.go:117] "RemoveContainer" containerID="f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57"
Sep 17 14:23:33.392756 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.392738    2565 scope.go:117] "RemoveContainer" containerID="f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57"
Sep 17 14:23:33.393016 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:23:33.392996    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57\": container with ID starting with f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57 not found: ID does not exist" containerID="f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57"
Sep 17 14:23:33.393074 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.393021    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57"} err="failed to get container status \"f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57\": rpc error: code = NotFound desc = could not find container \"f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57\": container with ID starting with f325ae1a6ae4f8b63b2704137366d4dbcfb13554fddbf9f9c5bfbea7b7386b57 not found: ID does not exist"
Sep 17 14:23:33.398470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.398446    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:33.422180 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.422154    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"]
Sep 17 14:23:33.427218 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.427192    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-7f7de479-7fddcb7f7d-v2blr"]
Sep 17 14:23:33.439207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.439184    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"]
Sep 17 14:23:33.442641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.442619    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29a9225f-f239-471d-a7be-19228d814c80-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:33.442641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.442639    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvm9b\" (UniqueName: \"kubernetes.io/projected/29a9225f-f239-471d-a7be-19228d814c80-kube-api-access-gvm9b\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:33.442743 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.442650    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7t7\" (UniqueName: \"kubernetes.io/projected/c1137789-bce2-4cbb-84a2-7747dfe73e29-kube-api-access-tq7t7\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:33.445879 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.445826    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-7f7de479-84b8898c9d-6tnhh"]
Sep 17 14:23:33.520834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.520802    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9996f7-e318-48ed-b23c-e31e8a11239f" path="/var/lib/kubelet/pods/0a9996f7-e318-48ed-b23c-e31e8a11239f/volumes"
Sep 17 14:23:33.521228 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.521214    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29a9225f-f239-471d-a7be-19228d814c80" path="/var/lib/kubelet/pods/29a9225f-f239-471d-a7be-19228d814c80/volumes"
Sep 17 14:23:33.521633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.521619    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1137789-bce2-4cbb-84a2-7747dfe73e29" path="/var/lib/kubelet/pods/c1137789-bce2-4cbb-84a2-7747dfe73e29/volumes"
Sep 17 14:23:33.679242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.679214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:33.870731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:33.870702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:34.237296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237197    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"]
Sep 17 14:23:34.237650 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237640    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1137789-bce2-4cbb-84a2-7747dfe73e29" containerName="payload-pre-processing"
Sep 17 14:23:34.237698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237655    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1137789-bce2-4cbb-84a2-7747dfe73e29" containerName="payload-pre-processing"
Sep 17 14:23:34.237698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237669    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9996f7-e318-48ed-b23c-e31e8a11239f" containerName="revoke-keys"
Sep 17 14:23:34.237698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237674    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9996f7-e318-48ed-b23c-e31e8a11239f" containerName="revoke-keys"
Sep 17 14:23:34.237698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237691    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29a9225f-f239-471d-a7be-19228d814c80" containerName="payload-processing"
Sep 17 14:23:34.237698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237697    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29a9225f-f239-471d-a7be-19228d814c80" containerName="payload-processing"
Sep 17 14:23:34.237857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237758    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1137789-bce2-4cbb-84a2-7747dfe73e29" containerName="payload-pre-processing"
Sep 17 14:23:34.237857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237766    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9996f7-e318-48ed-b23c-e31e8a11239f" containerName="revoke-keys"
Sep 17 14:23:34.237857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.237772    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29a9225f-f239-471d-a7be-19228d814c80" containerName="payload-processing"
Sep 17 14:23:34.240868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.240847    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.243617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.243595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1ecaca94-openshift-default-dockercfg-hgw2v\""
Sep 17 14:23:34.267534 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.267507    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"]
Sep 17 14:23:34.284284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.284231    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:34.353300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-data\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.353517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353339    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-istio-token\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.353517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-envoy\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.353517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353462    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bw5j\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-kube-api-access-9bw5j\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.353688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0876a65d-06dc-42b9-ab23-37432a56998f-istiod-ca-cert\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.353688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0876a65d-06dc-42b9-ab23-37432a56998f-istio-podinfo\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.353688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-credential-socket\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.353835 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353693    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-socket\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.353835 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.353732    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-certs\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.454738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.454704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0876a65d-06dc-42b9-ab23-37432a56998f-istio-podinfo\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.454938 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.454755    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-credential-socket\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.454938 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.454792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-socket\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.454938 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.454826    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-certs\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.455210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.455177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-data\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.455210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.455213    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-certs\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.455443 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.455287    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-istio-token\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.455443 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.455366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-credential-socket\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.455443 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.455386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-envoy\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.455443 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.455416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bw5j\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-kube-api-access-9bw5j\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.455669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.455495    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0876a65d-06dc-42b9-ab23-37432a56998f-istiod-ca-cert\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.455899 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.455880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-data\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.456152 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.456129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-socket\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.456234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.456225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0876a65d-06dc-42b9-ab23-37432a56998f-istiod-ca-cert\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.457707 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.457682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-envoy\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.457839 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.457820    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0876a65d-06dc-42b9-ab23-37432a56998f-istio-podinfo\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.470695 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.470674    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bw5j\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-kube-api-access-9bw5j\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.472151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.472131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-istio-token\") pod \"e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") " pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.509049 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.508961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:34.552179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.552150    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:34.731883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.731835    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:34.841717 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.841691    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"]
Sep 17 14:23:34.842224 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:23:34.842199    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0876a65d_06dc_42b9_ab23_37432a56998f.slice/crio-fa23269b403b26780566633371189a4d798274980379da74a265f73525f3e031 WatchSource:0}: Error finding container fa23269b403b26780566633371189a4d798274980379da74a265f73525f3e031: Status 404 returned error can't find the container with id fa23269b403b26780566633371189a4d798274980379da74a265f73525f3e031
Sep 17 14:23:34.844673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.844642    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:23:34.844740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.844714    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:23:34.844784 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:34.844744    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:23:35.388530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:35.388498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" event={"ID":"0876a65d-06dc-42b9-ab23-37432a56998f","Type":"ContainerStarted","Data":"658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2"}
Sep 17 14:23:35.388923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:35.388553    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" event={"ID":"0876a65d-06dc-42b9-ab23-37432a56998f","Type":"ContainerStarted","Data":"fa23269b403b26780566633371189a4d798274980379da74a265f73525f3e031"}
Sep 17 14:23:35.459678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:35.459620    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" podStartSLOduration=1.459603322 podStartE2EDuration="1.459603322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:23:35.458830542 +0000 UTC m=+1002.392335745" watchObservedRunningTime="2026-09-17 14:23:35.459603322 +0000 UTC m=+1002.393108515"
Sep 17 14:23:35.553321 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:35.553287    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:35.558848 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:35.558823    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:36.077848 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:36.077806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:36.394105 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:36.394012    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:36.394886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:36.394868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:23:36.574525 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:36.574476    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:37.236014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:37.235980    2565 patch_prober.go:28] interesting pod/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 17 14:23:37.236185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:37.236043    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:23:37.299682 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:37.299648    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:37.585293 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:37.585239    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:38.475597 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.475573    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:38.643126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.643073    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:23:38.814530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814499    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-data\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.814530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-socket\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.814752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814585    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-podinfo\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.814752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-token\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.814752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814646    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-credential-socket\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.814752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814674    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgt5p\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-kube-api-access-jgt5p\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.814752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814725    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420a4aa0-df86-4b75-bf0e-b73ac1956693-istiod-ca-cert\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.814752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814751    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-envoy\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.815087 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814757    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-data" (OuterVolumeSpecName: "istio-data") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:23:38.815087 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-certs\") pod \"420a4aa0-df86-4b75-bf0e-b73ac1956693\" (UID: \"420a4aa0-df86-4b75-bf0e-b73ac1956693\") "
Sep 17 14:23:38.815087 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814833    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:23:38.815087 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.814971    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:23:38.815087 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.815026    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/420a4aa0-df86-4b75-bf0e-b73ac1956693-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:23:38.815337 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.815180    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:23:38.815337 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.815328    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:38.815456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.815348    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/420a4aa0-df86-4b75-bf0e-b73ac1956693-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:38.815456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.815363    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:38.815456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.815375    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:38.815456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.815390    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:38.817021 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.816989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:23:38.817143 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.817068    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-token" (OuterVolumeSpecName: "istio-token") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:23:38.817143 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.817081    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:23:38.817143 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.817089    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-kube-api-access-jgt5p" (OuterVolumeSpecName: "kube-api-access-jgt5p") pod "420a4aa0-df86-4b75-bf0e-b73ac1956693" (UID: "420a4aa0-df86-4b75-bf0e-b73ac1956693"). InnerVolumeSpecName "kube-api-access-jgt5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:23:38.916447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.916368    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:38.916447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.916407    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:38.916447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.916421    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgt5p\" (UniqueName: \"kubernetes.io/projected/420a4aa0-df86-4b75-bf0e-b73ac1956693-kube-api-access-jgt5p\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:38.916447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:38.916434    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/420a4aa0-df86-4b75-bf0e-b73ac1956693-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:23:39.413467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.413433    2565 generic.go:356] "Generic (PLEG): container finished" podID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerID="80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b" exitCode=0
Sep 17 14:23:39.413672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.413485    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"
Sep 17 14:23:39.413672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.413514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" event={"ID":"420a4aa0-df86-4b75-bf0e-b73ac1956693","Type":"ContainerDied","Data":"80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b"}
Sep 17 14:23:39.413672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.413563    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5" event={"ID":"420a4aa0-df86-4b75-bf0e-b73ac1956693","Type":"ContainerDied","Data":"d91341200d2f214eecf37d7ec3b9617e4afb57203ff2b26e7bb1120ce8ad6840"}
Sep 17 14:23:39.413672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.413586    2565 scope.go:117] "RemoveContainer" containerID="80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b"
Sep 17 14:23:39.426285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.426246    2565 scope.go:117] "RemoveContainer" containerID="269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f"
Sep 17 14:23:39.434452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.434403    2565 scope.go:117] "RemoveContainer" containerID="80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b"
Sep 17 14:23:39.434737 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:23:39.434705    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b\": container with ID starting with 80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b not found: ID does not exist" containerID="80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b"
Sep 17 14:23:39.434848 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.434748    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b"} err="failed to get container status \"80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b\": rpc error: code = NotFound desc = could not find container \"80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b\": container with ID starting with 80d3fcaa104bb9500c9dc96674328df5a3cc195c6281d08b016fe312c2d8f16b not found: ID does not exist"
Sep 17 14:23:39.434848 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.434770    2565 scope.go:117] "RemoveContainer" containerID="269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f"
Sep 17 14:23:39.435103 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:23:39.435060    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f\": container with ID starting with 269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f not found: ID does not exist" containerID="269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f"
Sep 17 14:23:39.435241 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.435099    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f"} err="failed to get container status \"269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f\": rpc error: code = NotFound desc = could not find container \"269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f\": container with ID starting with 269c341c830416d8a1f6552aeb343e592eec916ff69b78b1b7fed39066811c3f not found: ID does not exist"
Sep 17 14:23:39.438137 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.438114    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"]
Sep 17 14:23:39.444697 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.444671    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-7f7de479-gw-openshift-default-6d45fb6c4d-th9k5"]
Sep 17 14:23:39.520790 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.520760    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" path="/var/lib/kubelet/pods/420a4aa0-df86-4b75-bf0e-b73ac1956693/volumes"
Sep 17 14:23:39.572681 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:39.572654    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:40.380465 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:40.380428    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:41.279837 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:41.279804    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:41.477472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:41.477440    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:41.593915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:41.593884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:41.880962 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:41.880876    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:41.985357 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:41.985321    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:42.005800 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.005770    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"]
Sep 17 14:23:42.006191 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.006178    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy"
Sep 17 14:23:42.006191 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.006192    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy"
Sep 17 14:23:42.006297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.006206    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy"
Sep 17 14:23:42.006297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.006212    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy"
Sep 17 14:23:42.006372 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.006300    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy"
Sep 17 14:23:42.006372 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.006307    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="420a4aa0-df86-4b75-bf0e-b73ac1956693" containerName="istio-proxy"
Sep 17 14:23:42.010579 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.010561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:42.014720 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.014701    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1ecaca94\""
Sep 17 14:23:42.030269 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.030237    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"]
Sep 17 14:23:42.049502 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.049476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8p8r\" (UniqueName: \"kubernetes.io/projected/f0ffb3b9-158d-420d-867e-32fba531524b-kube-api-access-n8p8r\") pod \"payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb\" (UID: \"f0ffb3b9-158d-420d-867e-32fba531524b\") " pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:42.049644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.049625    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ffb3b9-158d-420d-867e-32fba531524b-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb\" (UID: \"f0ffb3b9-158d-420d-867e-32fba531524b\") " pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:42.055754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.055732    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"]
Sep 17 14:23:42.058981 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.058966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:42.062805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.062787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1ecaca94-dockercfg-wqhrr\""
Sep 17 14:23:42.077444 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.077423    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"]
Sep 17 14:23:42.160328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.154828    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfznr\" (UniqueName: \"kubernetes.io/projected/4eaed19d-67f2-416a-8bcb-c92d29ed01be-kube-api-access-jfznr\") pod \"payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd\" (UID: \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:42.160328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.154929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8p8r\" (UniqueName: \"kubernetes.io/projected/f0ffb3b9-158d-420d-867e-32fba531524b-kube-api-access-n8p8r\") pod \"payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb\" (UID: \"f0ffb3b9-158d-420d-867e-32fba531524b\") " pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:42.160328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.154963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eaed19d-67f2-416a-8bcb-c92d29ed01be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd\" (UID: \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:42.160328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.155109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ffb3b9-158d-420d-867e-32fba531524b-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb\" (UID: \"f0ffb3b9-158d-420d-867e-32fba531524b\") " pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:42.160328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.156055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ffb3b9-158d-420d-867e-32fba531524b-plugins-config-volume\") pod \"payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb\" (UID: \"f0ffb3b9-158d-420d-867e-32fba531524b\") " pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:42.175934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.170408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8p8r\" (UniqueName: \"kubernetes.io/projected/f0ffb3b9-158d-420d-867e-32fba531524b-kube-api-access-n8p8r\") pod \"payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb\" (UID: \"f0ffb3b9-158d-420d-867e-32fba531524b\") " pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:42.255874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.255837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfznr\" (UniqueName: \"kubernetes.io/projected/4eaed19d-67f2-416a-8bcb-c92d29ed01be-kube-api-access-jfznr\") pod \"payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd\" (UID: \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:42.256037 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.255904    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eaed19d-67f2-416a-8bcb-c92d29ed01be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd\" (UID: \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:42.256601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.256581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eaed19d-67f2-416a-8bcb-c92d29ed01be-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd\" (UID: \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:42.265173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.265148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfznr\" (UniqueName: \"kubernetes.io/projected/4eaed19d-67f2-416a-8bcb-c92d29ed01be-kube-api-access-jfznr\") pod \"payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd\" (UID: \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:42.320731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.320707    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:42.354554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.354524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:42.368941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.368879    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:42.600805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.600779    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"]
Sep 17 14:23:42.601679 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:23:42.601654    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ffb3b9_158d_420d_867e_32fba531524b.slice/crio-6a8db00285e576fdc011e8e1fb87e14a23179cf6264610173e9f863b9ee2cc53 WatchSource:0}: Error finding container 6a8db00285e576fdc011e8e1fb87e14a23179cf6264610173e9f863b9ee2cc53: Status 404 returned error can't find the container with id 6a8db00285e576fdc011e8e1fb87e14a23179cf6264610173e9f863b9ee2cc53
Sep 17 14:23:42.684405 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:42.684373    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"]
Sep 17 14:23:42.686636 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:23:42.686613    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eaed19d_67f2_416a_8bcb_c92d29ed01be.slice/crio-ad0e440c7465880376fcc97e33f818ea695943fa10e7c524c8980221c1be48c5 WatchSource:0}: Error finding container ad0e440c7465880376fcc97e33f818ea695943fa10e7c524c8980221c1be48c5: Status 404 returned error can't find the container with id ad0e440c7465880376fcc97e33f818ea695943fa10e7c524c8980221c1be48c5
Sep 17 14:23:43.207427 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:43.207396    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:43.458675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:43.458583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb" event={"ID":"f0ffb3b9-158d-420d-867e-32fba531524b","Type":"ContainerStarted","Data":"35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873"}
Sep 17 14:23:43.458675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:43.458648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb" event={"ID":"f0ffb3b9-158d-420d-867e-32fba531524b","Type":"ContainerStarted","Data":"6a8db00285e576fdc011e8e1fb87e14a23179cf6264610173e9f863b9ee2cc53"}
Sep 17 14:23:43.458888 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:43.458753    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:43.460416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:43.460388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd" event={"ID":"4eaed19d-67f2-416a-8bcb-c92d29ed01be","Type":"ContainerStarted","Data":"7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb"}
Sep 17 14:23:43.460533 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:43.460428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd" event={"ID":"4eaed19d-67f2-416a-8bcb-c92d29ed01be","Type":"ContainerStarted","Data":"ad0e440c7465880376fcc97e33f818ea695943fa10e7c524c8980221c1be48c5"}
Sep 17 14:23:43.460533 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:43.460457    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:43.486951 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:43.486904    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb" podStartSLOduration=2.486888632 podStartE2EDuration="2.486888632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:23:43.48684425 +0000 UTC m=+1010.420349444" watchObservedRunningTime="2026-09-17 14:23:43.486888632 +0000 UTC m=+1010.420393826"
Sep 17 14:23:44.275825 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:44.275789    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:46.388581 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:46.388546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:23:54.469173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:54.469126    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:23:54.469838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:54.469806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:23:54.495654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:54.495611    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd" podStartSLOduration=12.495597056 podStartE2EDuration="12.495597056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:23:43.51647033 +0000 UTC m=+1010.449975522" watchObservedRunningTime="2026-09-17 14:23:54.495597056 +0000 UTC m=+1021.429102247"
Sep 17 14:23:58.891813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:23:58.891769    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:00.558310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:00.558238    2565 generic.go:356] "Generic (PLEG): container finished" podID="0876a65d-06dc-42b9-ab23-37432a56998f" containerID="658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2" exitCode=0
Sep 17 14:24:00.558709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:00.558283    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" event={"ID":"0876a65d-06dc-42b9-ab23-37432a56998f","Type":"ContainerDied","Data":"658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2"}
Sep 17 14:24:00.558805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:00.558789    2565 scope.go:117] "RemoveContainer" containerID="658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2"
Sep 17 14:24:00.559994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:00.559962    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:00.560119 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:00.560051    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:00.560119 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:00.560102    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:01.572780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:01.572732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" event={"ID":"0876a65d-06dc-42b9-ab23-37432a56998f","Type":"ContainerStarted","Data":"b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5"}
Sep 17 14:24:02.515302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:02.515267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:02.552431 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:02.552394    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:24:02.554184 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:02.553864    2565 patch_prober.go:28] interesting pod/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 17 14:24:02.554184 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:02.553920    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 17 14:24:03.552821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:03.552786    2565 patch_prober.go:28] interesting pod/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 17 14:24:03.553332 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:03.552861    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 17 14:24:04.215817 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:04.215781    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:04.553441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:04.553346    2565 patch_prober.go:28] interesting pod/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 17 14:24:04.553441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:04.553417    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 17 14:24:05.074480 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:05.074440    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:05.552908 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:05.552815    2565 patch_prober.go:28] interesting pod/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 17 14:24:05.553247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:05.553216    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 17 14:24:06.062580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:06.062537    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:06.394124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:06.394084    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:24:06.553338 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:06.553304    2565 patch_prober.go:28] interesting pod/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Sep 17 14:24:06.553605 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:06.553575    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Sep 17 14:24:06.981705 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:06.981670    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:07.557429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:07.557399    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:24:07.558715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:07.558693    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:24:07.759305 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:07.759274    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:07.880539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:07.880506    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"]
Sep 17 14:24:07.883149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:07.883127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"
Sep 17 14:24:07.885783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:07.885763    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:24:07.886233 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:07.886207    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:24:07.897276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:07.897228    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"]
Sep 17 14:24:08.041738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:08.041707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62caaf02-9d81-47df-973f-fe045dfdb59a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml\" (UID: \"62caaf02-9d81-47df-973f-fe045dfdb59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"
Sep 17 14:24:08.142505 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:08.142418    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62caaf02-9d81-47df-973f-fe045dfdb59a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml\" (UID: \"62caaf02-9d81-47df-973f-fe045dfdb59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"
Sep 17 14:24:08.143013 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:08.142993    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62caaf02-9d81-47df-973f-fe045dfdb59a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml\" (UID: \"62caaf02-9d81-47df-973f-fe045dfdb59a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"
Sep 17 14:24:08.193418 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:08.193390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"
Sep 17 14:24:08.426081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:08.425998    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:08.508321 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:08.508244    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"]
Sep 17 14:24:08.510743 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:24:08.510714    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62caaf02_9d81_47df_973f_fe045dfdb59a.slice/crio-332570d068e1f590c3fd089655f43746f4b161f6c9d110a754641571119f73cc WatchSource:0}: Error finding container 332570d068e1f590c3fd089655f43746f4b161f6c9d110a754641571119f73cc: Status 404 returned error can't find the container with id 332570d068e1f590c3fd089655f43746f4b161f6c9d110a754641571119f73cc
Sep 17 14:24:08.579145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:08.579075    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:08.614433 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:08.614405    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml" event={"ID":"62caaf02-9d81-47df-973f-fe045dfdb59a","Type":"ContainerStarted","Data":"332570d068e1f590c3fd089655f43746f4b161f6c9d110a754641571119f73cc"}
Sep 17 14:24:09.203625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:09.203594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:09.621556 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:09.621522    2565 generic.go:356] "Generic (PLEG): container finished" podID="62caaf02-9d81-47df-973f-fe045dfdb59a" containerID="d9e7b7db15cb4b56087433d1b3b8acbe1705cf7c84ed985c89f5053b6e5b928b" exitCode=0
Sep 17 14:24:09.621951 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:09.621577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml" event={"ID":"62caaf02-9d81-47df-973f-fe045dfdb59a","Type":"ContainerDied","Data":"d9e7b7db15cb4b56087433d1b3b8acbe1705cf7c84ed985c89f5053b6e5b928b"}
Sep 17 14:24:10.050499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.050349    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:10.199523 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.199482    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"]
Sep 17 14:24:10.202071 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.202054    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"
Sep 17 14:24:10.232508 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.232483    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"]
Sep 17 14:24:10.265192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.265153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f2ef1-8c9d-42ca-afde-49f98f741189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-86c464-6wkhq\" (UID: \"692f2ef1-8c9d-42ca-afde-49f98f741189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"
Sep 17 14:24:10.366124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.366098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f2ef1-8c9d-42ca-afde-49f98f741189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-86c464-6wkhq\" (UID: \"692f2ef1-8c9d-42ca-afde-49f98f741189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"
Sep 17 14:24:10.366707 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.366688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f2ef1-8c9d-42ca-afde-49f98f741189-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-86c464-6wkhq\" (UID: \"692f2ef1-8c9d-42ca-afde-49f98f741189\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"
Sep 17 14:24:10.512290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.512244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"
Sep 17 14:24:10.779880 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.779856    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"
Sep 17 14:24:10.786630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.786605    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"]
Sep 17 14:24:10.787707 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:24:10.787677    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692f2ef1_8c9d_42ca_afde_49f98f741189.slice/crio-336ea45290efd500faee3e9a6134217464cb07703cd1bbdbe58eb77cba993b73 WatchSource:0}: Error finding container 336ea45290efd500faee3e9a6134217464cb07703cd1bbdbe58eb77cba993b73: Status 404 returned error can't find the container with id 336ea45290efd500faee3e9a6134217464cb07703cd1bbdbe58eb77cba993b73
Sep 17 14:24:10.872082 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.872057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62caaf02-9d81-47df-973f-fe045dfdb59a-maas-api-service-ca\") pod \"62caaf02-9d81-47df-973f-fe045dfdb59a\" (UID: \"62caaf02-9d81-47df-973f-fe045dfdb59a\") "
Sep 17 14:24:10.872407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.872387    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62caaf02-9d81-47df-973f-fe045dfdb59a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62caaf02-9d81-47df-973f-fe045dfdb59a" (UID: "62caaf02-9d81-47df-973f-fe045dfdb59a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:10.973493 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:10.973470    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62caaf02-9d81-47df-973f-fe045dfdb59a-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:11.273217 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:11.273133    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:11.635158 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:11.635122    2565 generic.go:356] "Generic (PLEG): container finished" podID="692f2ef1-8c9d-42ca-afde-49f98f741189" containerID="b423fd85f357ca1a26e3e9c9f985d55e470c737fe1e5a827b4ea102c55a8f794" exitCode=0
Sep 17 14:24:11.635375 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:11.635207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq" event={"ID":"692f2ef1-8c9d-42ca-afde-49f98f741189","Type":"ContainerDied","Data":"b423fd85f357ca1a26e3e9c9f985d55e470c737fe1e5a827b4ea102c55a8f794"}
Sep 17 14:24:11.635375 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:11.635273    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq" event={"ID":"692f2ef1-8c9d-42ca-afde-49f98f741189","Type":"ContainerStarted","Data":"336ea45290efd500faee3e9a6134217464cb07703cd1bbdbe58eb77cba993b73"}
Sep 17 14:24:11.636751 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:11.636726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml" event={"ID":"62caaf02-9d81-47df-973f-fe045dfdb59a","Type":"ContainerDied","Data":"332570d068e1f590c3fd089655f43746f4b161f6c9d110a754641571119f73cc"}
Sep 17 14:24:11.636867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:11.636765    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="332570d068e1f590c3fd089655f43746f4b161f6c9d110a754641571119f73cc"
Sep 17 14:24:11.636867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:11.636771    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"
Sep 17 14:24:12.345748 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:12.345718    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:12.785855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:12.785824    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"
Sep 17 14:24:12.893022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:12.892933    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f2ef1-8c9d-42ca-afde-49f98f741189-maas-api-service-ca\") pod \"692f2ef1-8c9d-42ca-afde-49f98f741189\" (UID: \"692f2ef1-8c9d-42ca-afde-49f98f741189\") "
Sep 17 14:24:12.893324 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:12.893295    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f2ef1-8c9d-42ca-afde-49f98f741189-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "692f2ef1-8c9d-42ca-afde-49f98f741189" (UID: "692f2ef1-8c9d-42ca-afde-49f98f741189"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:12.994184 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:12.994137    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f2ef1-8c9d-42ca-afde-49f98f741189-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:13.384102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.384067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"]
Sep 17 14:24:13.384530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.384515    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62caaf02-9d81-47df-973f-fe045dfdb59a" containerName="revoke-keys"
Sep 17 14:24:13.384530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.384530    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="62caaf02-9d81-47df-973f-fe045dfdb59a" containerName="revoke-keys"
Sep 17 14:24:13.384614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.384554    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692f2ef1-8c9d-42ca-afde-49f98f741189" containerName="revoke-keys"
Sep 17 14:24:13.384614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.384560    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="692f2ef1-8c9d-42ca-afde-49f98f741189" containerName="revoke-keys"
Sep 17 14:24:13.384680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.384626    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="62caaf02-9d81-47df-973f-fe045dfdb59a" containerName="revoke-keys"
Sep 17 14:24:13.384680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.384638    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="692f2ef1-8c9d-42ca-afde-49f98f741189" containerName="revoke-keys"
Sep 17 14:24:13.386819 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.386797    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.393498 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.393471    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2e1b82\"/\"openshift-service-ca.crt\""
Sep 17 14:24:13.394162 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.394139    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2e1b82\"/\"kube-root-ca.crt\""
Sep 17 14:24:13.394286 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.394207    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2e1b82\"/\"default-dockercfg-p79fr\""
Sep 17 14:24:13.394521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.394504    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2e1b82\"/\"test-model-2e1b82-kserve-self-signed-certs\""
Sep 17 14:24:13.399564 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.399544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-model-cache\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.399651 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.399578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dgr4\" (UniqueName: \"kubernetes.io/projected/66468531-68b5-4298-b30a-80770b511f95-kube-api-access-7dgr4\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.399692 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.399662    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-home\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.399692 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.399686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468531-68b5-4298-b30a-80770b511f95-tls-certs\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.399758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.399707    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-tmp-dir\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.399758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.399731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-dshm\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.428991 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.428964    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"]
Sep 17 14:24:13.501138 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501100    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-home\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.501348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501144    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468531-68b5-4298-b30a-80770b511f95-tls-certs\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.501348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501174    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-tmp-dir\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.501348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-dshm\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.501348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-model-cache\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.501348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dgr4\" (UniqueName: \"kubernetes.io/projected/66468531-68b5-4298-b30a-80770b511f95-kube-api-access-7dgr4\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.501628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501583    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-home\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.501628 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-tmp-dir\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.501731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.501661    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-model-cache\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.503615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.503587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-dshm\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.503907 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.503887    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468531-68b5-4298-b30a-80770b511f95-tls-certs\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.513549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.513526    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dgr4\" (UniqueName: \"kubernetes.io/projected/66468531-68b5-4298-b30a-80770b511f95-kube-api-access-7dgr4\") pod \"test-model-2e1b82-kserve-77465b55bf-qp4tz\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") " pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:13.607097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.607066    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:13.651824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.651728    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq" event={"ID":"692f2ef1-8c9d-42ca-afde-49f98f741189","Type":"ContainerDied","Data":"336ea45290efd500faee3e9a6134217464cb07703cd1bbdbe58eb77cba993b73"}
Sep 17 14:24:13.652007 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.651835    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="336ea45290efd500faee3e9a6134217464cb07703cd1bbdbe58eb77cba993b73"
Sep 17 14:24:13.652007 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.651751    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"
Sep 17 14:24:13.696568 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:13.696537    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:14.001862 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:14.001839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"]
Sep 17 14:24:14.003970 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:24:14.003947    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66468531_68b5_4298_b30a_80770b511f95.slice/crio-dcd81d59b5b8f016b9134e6e10267601cbc25034510f3149c959d46b0c7ef833 WatchSource:0}: Error finding container dcd81d59b5b8f016b9134e6e10267601cbc25034510f3149c959d46b0c7ef833: Status 404 returned error can't find the container with id dcd81d59b5b8f016b9134e6e10267601cbc25034510f3149c959d46b0c7ef833
Sep 17 14:24:14.288345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:14.288237    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:14.663912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:14.663874    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz" event={"ID":"66468531-68b5-4298-b30a-80770b511f95","Type":"ContainerStarted","Data":"2ae0622b98081d1c19c21b6cd62aff452c039a3de7a07499940c0568592204ae"}
Sep 17 14:24:14.664351 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:14.663923    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz" event={"ID":"66468531-68b5-4298-b30a-80770b511f95","Type":"ContainerStarted","Data":"dcd81d59b5b8f016b9134e6e10267601cbc25034510f3149c959d46b0c7ef833"}
Sep 17 14:24:14.691708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:14.691642    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz" podStartSLOduration=1.69162496 podStartE2EDuration="1.69162496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:14.690008422 +0000 UTC m=+1041.623513615" watchObservedRunningTime="2026-09-17 14:24:14.69162496 +0000 UTC m=+1041.625130150"
Sep 17 14:24:15.095606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:15.095561    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:16.067759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:16.067726    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:16.846844 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:16.846813    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:19.286155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.286120    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:19.324921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.324890    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"]
Sep 17 14:24:19.325833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.325799    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb" podUID="f0ffb3b9-158d-420d-867e-32fba531524b" containerName="payload-processing" containerID="cri-o://35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873" gracePeriod=30
Sep 17 14:24:19.330377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.330352    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"]
Sep 17 14:24:19.330766 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.330718    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd" podUID="4eaed19d-67f2-416a-8bcb-c92d29ed01be" containerName="payload-pre-processing" containerID="cri-o://7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb" gracePeriod=30
Sep 17 14:24:19.379214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.379181    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:19.459848 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.459812    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:19.543548 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.543492    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"]
Sep 17 14:24:19.547439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.547416    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1ecaca94-f6xml"]
Sep 17 14:24:19.553002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.552974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:19.596699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.596662    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:24:19.613699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.613680    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:24:19.624786 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.624767    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:19.695398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.679025    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ffb3b9-158d-420d-867e-32fba531524b-plugins-config-volume\") pod \"f0ffb3b9-158d-420d-867e-32fba531524b\" (UID: \"f0ffb3b9-158d-420d-867e-32fba531524b\") "
Sep 17 14:24:19.695398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.679098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8p8r\" (UniqueName: \"kubernetes.io/projected/f0ffb3b9-158d-420d-867e-32fba531524b-kube-api-access-n8p8r\") pod \"f0ffb3b9-158d-420d-867e-32fba531524b\" (UID: \"f0ffb3b9-158d-420d-867e-32fba531524b\") "
Sep 17 14:24:19.695398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.679157    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eaed19d-67f2-416a-8bcb-c92d29ed01be-plugins-config-volume\") pod \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\" (UID: \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\") "
Sep 17 14:24:19.695398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.679282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfznr\" (UniqueName: \"kubernetes.io/projected/4eaed19d-67f2-416a-8bcb-c92d29ed01be-kube-api-access-jfznr\") pod \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\" (UID: \"4eaed19d-67f2-416a-8bcb-c92d29ed01be\") "
Sep 17 14:24:19.695398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.680586    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0ffb3b9-158d-420d-867e-32fba531524b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f0ffb3b9-158d-420d-867e-32fba531524b" (UID: "f0ffb3b9-158d-420d-867e-32fba531524b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:19.695398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.680992    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eaed19d-67f2-416a-8bcb-c92d29ed01be-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4eaed19d-67f2-416a-8bcb-c92d29ed01be" (UID: "4eaed19d-67f2-416a-8bcb-c92d29ed01be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:19.695398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.691320    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eaed19d-67f2-416a-8bcb-c92d29ed01be-kube-api-access-jfznr" (OuterVolumeSpecName: "kube-api-access-jfznr") pod "4eaed19d-67f2-416a-8bcb-c92d29ed01be" (UID: "4eaed19d-67f2-416a-8bcb-c92d29ed01be"). InnerVolumeSpecName "kube-api-access-jfznr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:19.695398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.691418    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ffb3b9-158d-420d-867e-32fba531524b-kube-api-access-n8p8r" (OuterVolumeSpecName: "kube-api-access-n8p8r") pod "f0ffb3b9-158d-420d-867e-32fba531524b" (UID: "f0ffb3b9-158d-420d-867e-32fba531524b"). InnerVolumeSpecName "kube-api-access-n8p8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:19.704616 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.704579    2565 generic.go:356] "Generic (PLEG): container finished" podID="f0ffb3b9-158d-420d-867e-32fba531524b" containerID="35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873" exitCode=0
Sep 17 14:24:19.704809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.704636    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"
Sep 17 14:24:19.704809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.704662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb" event={"ID":"f0ffb3b9-158d-420d-867e-32fba531524b","Type":"ContainerDied","Data":"35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873"}
Sep 17 14:24:19.704809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.704725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb" event={"ID":"f0ffb3b9-158d-420d-867e-32fba531524b","Type":"ContainerDied","Data":"6a8db00285e576fdc011e8e1fb87e14a23179cf6264610173e9f863b9ee2cc53"}
Sep 17 14:24:19.704809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.704750    2565 scope.go:117] "RemoveContainer" containerID="35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873"
Sep 17 14:24:19.706191 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.706167    2565 generic.go:356] "Generic (PLEG): container finished" podID="4eaed19d-67f2-416a-8bcb-c92d29ed01be" containerID="7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb" exitCode=0
Sep 17 14:24:19.706315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.706234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd" event={"ID":"4eaed19d-67f2-416a-8bcb-c92d29ed01be","Type":"ContainerDied","Data":"7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb"}
Sep 17 14:24:19.706315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.706230    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"
Sep 17 14:24:19.706315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.706289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd" event={"ID":"4eaed19d-67f2-416a-8bcb-c92d29ed01be","Type":"ContainerDied","Data":"ad0e440c7465880376fcc97e33f818ea695943fa10e7c524c8980221c1be48c5"}
Sep 17 14:24:19.715608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.715592    2565 scope.go:117] "RemoveContainer" containerID="35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873"
Sep 17 14:24:19.715921 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:24:19.715903    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873\": container with ID starting with 35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873 not found: ID does not exist" containerID="35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873"
Sep 17 14:24:19.715995 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.715928    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873"} err="failed to get container status \"35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873\": rpc error: code = NotFound desc = could not find container \"35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873\": container with ID starting with 35e3ca3960bce43e05687d82fe390ac77e18bd2de7501a57488fa0a5a06ae873 not found: ID does not exist"
Sep 17 14:24:19.715995 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.715944    2565 scope.go:117] "RemoveContainer" containerID="7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb"
Sep 17 14:24:19.724245 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.724227    2565 scope.go:117] "RemoveContainer" containerID="7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb"
Sep 17 14:24:19.724554 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:24:19.724534    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb\": container with ID starting with 7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb not found: ID does not exist" containerID="7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb"
Sep 17 14:24:19.724619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.724560    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb"} err="failed to get container status \"7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb\": rpc error: code = NotFound desc = could not find container \"7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb\": container with ID starting with 7e3da9920dba45b9c776507003621481c6263fecccfc0f05830c1b06b912d0eb not found: ID does not exist"
Sep 17 14:24:19.730897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.730872    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"]
Sep 17 14:24:19.736624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.736599    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1ecaca94-656f8d7c4f-qzjdb"]
Sep 17 14:24:19.748874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.748852    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"]
Sep 17 14:24:19.755232 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.755209    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1ecaca94-7879cd6b94-7flzd"]
Sep 17 14:24:19.780920 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.780898    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfznr\" (UniqueName: \"kubernetes.io/projected/4eaed19d-67f2-416a-8bcb-c92d29ed01be-kube-api-access-jfznr\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:19.781002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.780921    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0ffb3b9-158d-420d-867e-32fba531524b-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:19.781002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.780937    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8p8r\" (UniqueName: \"kubernetes.io/projected/f0ffb3b9-158d-420d-867e-32fba531524b-kube-api-access-n8p8r\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:19.781002 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.780948    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eaed19d-67f2-416a-8bcb-c92d29ed01be-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:19.970730 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:19.970695    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:20.659512 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:20.659433    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"]
Sep 17 14:24:20.659882 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:20.659694    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy" containerID="cri-o://b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5" gracePeriod=30
Sep 17 14:24:20.694982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:20.694953    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:20.830626 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:20.830594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:21.115995 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.115965    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:21.520561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.520480    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eaed19d-67f2-416a-8bcb-c92d29ed01be" path="/var/lib/kubelet/pods/4eaed19d-67f2-416a-8bcb-c92d29ed01be/volumes"
Sep 17 14:24:21.520880 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.520866    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62caaf02-9d81-47df-973f-fe045dfdb59a" path="/var/lib/kubelet/pods/62caaf02-9d81-47df-973f-fe045dfdb59a/volumes"
Sep 17 14:24:21.521204 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.521192    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ffb3b9-158d-420d-867e-32fba531524b" path="/var/lib/kubelet/pods/f0ffb3b9-158d-420d-867e-32fba531524b/volumes"
Sep 17 14:24:21.538970 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.538936    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"]
Sep 17 14:24:21.539843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.539814    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eaed19d-67f2-416a-8bcb-c92d29ed01be" containerName="payload-pre-processing"
Sep 17 14:24:21.539843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.539844    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eaed19d-67f2-416a-8bcb-c92d29ed01be" containerName="payload-pre-processing"
Sep 17 14:24:21.540025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.539914    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ffb3b9-158d-420d-867e-32fba531524b" containerName="payload-processing"
Sep 17 14:24:21.540025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.539924    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ffb3b9-158d-420d-867e-32fba531524b" containerName="payload-processing"
Sep 17 14:24:21.540144 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.540130    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eaed19d-67f2-416a-8bcb-c92d29ed01be" containerName="payload-pre-processing"
Sep 17 14:24:21.540199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.540156    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ffb3b9-158d-420d-867e-32fba531524b" containerName="payload-processing"
Sep 17 14:24:21.544459 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.544439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.547376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.547357    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2f37280d-openshift-default-dockercfg-pgpk6\""
Sep 17 14:24:21.560820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.560795    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"]
Sep 17 14:24:21.573936 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.573915    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:21.601268 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601230    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-envoy\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.601370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34058834-c753-4e81-8391-066fcd936639-istiod-ca-cert\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.601370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-data\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.601495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-credential-socket\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.601495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-socket\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.601597 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-istio-token\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.601641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601629    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34058834-c753-4e81-8391-066fcd936639-istio-podinfo\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.601693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n684p\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-kube-api-access-n684p\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.601747 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.601702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-certs\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.702574 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702536    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-envoy\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.702574 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34058834-c753-4e81-8391-066fcd936639-istiod-ca-cert\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702622    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-data\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702644    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-credential-socket\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702661    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-socket\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-istio-token\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34058834-c753-4e81-8391-066fcd936639-istio-podinfo\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n684p\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-kube-api-access-n684p\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.702836    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-certs\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.703068    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-credential-socket\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.703131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-data\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.703192    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-certs\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.703449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-socket\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.703796 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.703774    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34058834-c753-4e81-8391-066fcd936639-istiod-ca-cert\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.704923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.704899    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-envoy\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.705493 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.705469    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34058834-c753-4e81-8391-066fcd936639-istio-podinfo\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.713404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.713373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-istio-token\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.713520 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.713486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n684p\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-kube-api-access-n684p\") pod \"e2e-derive-2f37280d-openshift-default-647575f4d-btg4c\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") " pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.857197 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.857161    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:21.963102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:21.963073    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:22.095293 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.095236    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:22.151573 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.151549    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"]
Sep 17 14:24:22.154293 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:24:22.154242    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34058834_c753_4e81_8391_066fcd936639.slice/crio-a36d148407ab46fd3b6cf10744ab784928fe0b250589d7d4abae09b51d5dc979 WatchSource:0}: Error finding container a36d148407ab46fd3b6cf10744ab784928fe0b250589d7d4abae09b51d5dc979: Status 404 returned error can't find the container with id a36d148407ab46fd3b6cf10744ab784928fe0b250589d7d4abae09b51d5dc979
Sep 17 14:24:22.156742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.156708    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:22.156847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.156775    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:22.156847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.156806    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:22.560334 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.560237    2565 patch_prober.go:28] interesting pod/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:24:22.560334 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.560317    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:24:22.729554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.729521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" event={"ID":"34058834-c753-4e81-8391-066fcd936639","Type":"ContainerStarted","Data":"e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1"}
Sep 17 14:24:22.729942 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.729573    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" event={"ID":"34058834-c753-4e81-8391-066fcd936639","Type":"ContainerStarted","Data":"a36d148407ab46fd3b6cf10744ab784928fe0b250589d7d4abae09b51d5dc979"}
Sep 17 14:24:22.755234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.754899    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" podStartSLOduration=1.7548763059999999 podStartE2EDuration="1.754876306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:22.750497513 +0000 UTC m=+1049.684002708" watchObservedRunningTime="2026-09-17 14:24:22.754876306 +0000 UTC m=+1049.688381499"
Sep 17 14:24:22.816085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.816004    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:22.858105 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.857840    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:22.896544 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:22.896501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:23.010641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.010555    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:23.077218 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.077185    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"]
Sep 17 14:24:23.080895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.080868    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-86c464-6wkhq"]
Sep 17 14:24:23.115969 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.115941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:23.216278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.215735    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:23.435700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.435629    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:23.520649 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.520616    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692f2ef1-8c9d-42ca-afde-49f98f741189" path="/var/lib/kubelet/pods/692f2ef1-8c9d-42ca-afde-49f98f741189/volumes"
Sep 17 14:24:23.697271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.697178    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:23.697271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.697227    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:23.709376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.709349    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:23.748310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.748279    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:24:23.863328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:23.863294    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:24.277418 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:24.277385    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:24.604018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:24.603977    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:24.720603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:24.720570    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:24.743346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:24.743319    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:24.744173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:24.744152    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:25.330113 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:25.330081    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:25.913210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:25.913186    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:24:26.056716 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.056624    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0876a65d-06dc-42b9-ab23-37432a56998f-istiod-ca-cert\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.056716 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.056678    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-data\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.056929 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.056755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-istio-token\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.056929 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.056806    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-certs\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.056929 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.056881    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-credential-socket\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.057099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.056920    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:26.057099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.056946    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0876a65d-06dc-42b9-ab23-37432a56998f-istio-podinfo\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.057099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-envoy\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.057099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057025    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-socket\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.057099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057078    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bw5j\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-kube-api-access-9bw5j\") pod \"0876a65d-06dc-42b9-ab23-37432a56998f\" (UID: \"0876a65d-06dc-42b9-ab23-37432a56998f\") "
Sep 17 14:24:26.057099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057085    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:26.057451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057115    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0876a65d-06dc-42b9-ab23-37432a56998f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:26.057451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057335    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:26.057561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:26.057561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057500    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0876a65d-06dc-42b9-ab23-37432a56998f-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.057561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057519    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.057561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057531    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.057561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.057544    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.059428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.059405    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:24:26.059553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.059455    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-kube-api-access-9bw5j" (OuterVolumeSpecName: "kube-api-access-9bw5j") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "kube-api-access-9bw5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:26.059553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.059482    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:24:26.059806 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.059779    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0876a65d-06dc-42b9-ab23-37432a56998f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0876a65d-06dc-42b9-ab23-37432a56998f" (UID: "0876a65d-06dc-42b9-ab23-37432a56998f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:24:26.158500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.158463    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.158500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.158493    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0876a65d-06dc-42b9-ab23-37432a56998f-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.158500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.158506    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bw5j\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-kube-api-access-9bw5j\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.158734 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.158519    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0876a65d-06dc-42b9-ab23-37432a56998f-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.158734 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.158530    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0876a65d-06dc-42b9-ab23-37432a56998f-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:26.167975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.167947    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:26.757167 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.757130    2565 generic.go:356] "Generic (PLEG): container finished" podID="0876a65d-06dc-42b9-ab23-37432a56998f" containerID="b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5" exitCode=0
Sep 17 14:24:26.757676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.757173    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"
Sep 17 14:24:26.757676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.757208    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" event={"ID":"0876a65d-06dc-42b9-ab23-37432a56998f","Type":"ContainerDied","Data":"b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5"}
Sep 17 14:24:26.757676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.757276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48" event={"ID":"0876a65d-06dc-42b9-ab23-37432a56998f","Type":"ContainerDied","Data":"fa23269b403b26780566633371189a4d798274980379da74a265f73525f3e031"}
Sep 17 14:24:26.757676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.757305    2565 scope.go:117] "RemoveContainer" containerID="b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5"
Sep 17 14:24:26.767215 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.767197    2565 scope.go:117] "RemoveContainer" containerID="658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2"
Sep 17 14:24:26.775518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.775502    2565 scope.go:117] "RemoveContainer" containerID="b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5"
Sep 17 14:24:26.775778 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:24:26.775758    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5\": container with ID starting with b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5 not found: ID does not exist" containerID="b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5"
Sep 17 14:24:26.775824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.775788    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5"} err="failed to get container status \"b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5\": rpc error: code = NotFound desc = could not find container \"b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5\": container with ID starting with b9374bf51cda055bcd357fe40cf52ac16c843c08914fd9200a8690bcd74b86c5 not found: ID does not exist"
Sep 17 14:24:26.775824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.775805    2565 scope.go:117] "RemoveContainer" containerID="658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2"
Sep 17 14:24:26.776040 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:24:26.776026    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2\": container with ID starting with 658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2 not found: ID does not exist" containerID="658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2"
Sep 17 14:24:26.776081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.776045    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2"} err="failed to get container status \"658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2\": rpc error: code = NotFound desc = could not find container \"658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2\": container with ID starting with 658827da8524e442c9694b555464844d35dc0a8d3eac64982d5f5995ffe033e2 not found: ID does not exist"
Sep 17 14:24:26.781097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.781075    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"]
Sep 17 14:24:26.786553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.786533    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1ecaca94-openshift-default-b9c7455fd-n6q48"]
Sep 17 14:24:26.893586 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:26.893549    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:27.519812 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:27.519779    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" path="/var/lib/kubelet/pods/0876a65d-06dc-42b9-ab23-37432a56998f/volumes"
Sep 17 14:24:27.783539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:27.783461    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:27.881811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:27.881780    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:27.953706 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:27.953672    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:28.300629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.300594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:28.407663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.407633    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:28.442650 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.442617    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"]
Sep 17 14:24:28.443215 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.443191    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy"
Sep 17 14:24:28.443215 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.443214    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy"
Sep 17 14:24:28.443412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.443233    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy"
Sep 17 14:24:28.443412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.443243    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy"
Sep 17 14:24:28.443412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.443359    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy"
Sep 17 14:24:28.443412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.443370    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0876a65d-06dc-42b9-ab23-37432a56998f" containerName="istio-proxy"
Sep 17 14:24:28.447900 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.447883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:28.457750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.457729    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-2f37280d\""
Sep 17 14:24:28.494182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.494149    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"]
Sep 17 14:24:28.585321 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.585287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1339c6df-9240-4fb6-89d6-f6820ab45d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz\" (UID: \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:28.585527 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.585448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xt6\" (UniqueName: \"kubernetes.io/projected/1339c6df-9240-4fb6-89d6-f6820ab45d8c-kube-api-access-v7xt6\") pod \"payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz\" (UID: \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:28.622037 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.621997    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:28.686714 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.686675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xt6\" (UniqueName: \"kubernetes.io/projected/1339c6df-9240-4fb6-89d6-f6820ab45d8c-kube-api-access-v7xt6\") pod \"payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz\" (UID: \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:28.687322 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.687292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1339c6df-9240-4fb6-89d6-f6820ab45d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz\" (UID: \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:28.687897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.687871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1339c6df-9240-4fb6-89d6-f6820ab45d8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz\" (UID: \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:28.698850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.698826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xt6\" (UniqueName: \"kubernetes.io/projected/1339c6df-9240-4fb6-89d6-f6820ab45d8c-kube-api-access-v7xt6\") pod \"payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz\" (UID: \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:28.758801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.758775    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:28.887325 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:28.887228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:29.055811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:29.055776    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"]
Sep 17 14:24:29.057551 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:24:29.057527    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1339c6df_9240_4fb6_89d6_f6820ab45d8c.slice/crio-7e48b7a3308899ad811e762d92dcc64d0a6add8600bb27a02a6b6428f9c58def WatchSource:0}: Error finding container 7e48b7a3308899ad811e762d92dcc64d0a6add8600bb27a02a6b6428f9c58def: Status 404 returned error can't find the container with id 7e48b7a3308899ad811e762d92dcc64d0a6add8600bb27a02a6b6428f9c58def
Sep 17 14:24:29.765481 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:29.765433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:29.777054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:29.777020    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz" event={"ID":"1339c6df-9240-4fb6-89d6-f6820ab45d8c","Type":"ContainerStarted","Data":"41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888"}
Sep 17 14:24:29.777188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:29.777073    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz" event={"ID":"1339c6df-9240-4fb6-89d6-f6820ab45d8c","Type":"ContainerStarted","Data":"7e48b7a3308899ad811e762d92dcc64d0a6add8600bb27a02a6b6428f9c58def"}
Sep 17 14:24:29.777279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:29.777241    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:29.800082 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:29.800037    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz" podStartSLOduration=1.800021672 podStartE2EDuration="1.800021672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:24:29.795792596 +0000 UTC m=+1056.729297788" watchObservedRunningTime="2026-09-17 14:24:29.800021672 +0000 UTC m=+1056.733526864"
Sep 17 14:24:38.129353 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:38.129315    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:40.785765 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:40.785728    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:24:45.895329 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:45.895295    2565 generic.go:356] "Generic (PLEG): container finished" podID="34058834-c753-4e81-8391-066fcd936639" containerID="e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1" exitCode=0
Sep 17 14:24:45.895329 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:45.895338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" event={"ID":"34058834-c753-4e81-8391-066fcd936639","Type":"ContainerDied","Data":"e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1"}
Sep 17 14:24:45.895772 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:45.895758    2565 scope.go:117] "RemoveContainer" containerID="e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1"
Sep 17 14:24:45.896945 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:45.896916    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:45.897008 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:45.896984    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:45.897045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:45.897016    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:24:46.903990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:46.903811    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" event={"ID":"34058834-c753-4e81-8391-066fcd936639","Type":"ContainerStarted","Data":"a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7"}
Sep 17 14:24:47.857440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:47.857389    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:47.857743 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:47.857719    2565 patch_prober.go:28] interesting pod/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:24:47.857940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:47.857914    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 17 14:24:48.858198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:48.858140    2565 patch_prober.go:28] interesting pod/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:24:48.858742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:48.858229    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 17 14:24:49.858347 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:49.858303    2565 patch_prober.go:28] interesting pod/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:24:49.858857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:49.858375    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 17 14:24:50.173303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:50.173186    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:50.858652 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:50.858143    2565 patch_prober.go:28] interesting pod/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:24:50.858652 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:50.858210    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 17 14:24:51.084479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:51.084444    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:51.857569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:51.857522    2565 patch_prober.go:28] interesting pod/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Sep 17 14:24:51.857763 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:51.857596    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Sep 17 14:24:52.058369 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:52.058337    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:52.213049 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:52.212962    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:52.862086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:52.862054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:52.862363 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:52.862342    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:52.863026 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:52.863011    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:24:52.946298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:52.946267    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:53.871240 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:53.871166    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:54.470414 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.470373    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"]
Sep 17 14:24:54.473647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.473619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"
Sep 17 14:24:54.476811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.476786    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:24:54.476811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.476811    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:24:54.492458 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.492435    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"]
Sep 17 14:24:54.560623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.560581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10f6b0-3830-4c87-a63e-2cb3350070df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl\" (UID: \"5f10f6b0-3830-4c87-a63e-2cb3350070df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"
Sep 17 14:24:54.661852 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.661815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10f6b0-3830-4c87-a63e-2cb3350070df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl\" (UID: \"5f10f6b0-3830-4c87-a63e-2cb3350070df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"
Sep 17 14:24:54.662494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.662468    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10f6b0-3830-4c87-a63e-2cb3350070df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl\" (UID: \"5f10f6b0-3830-4c87-a63e-2cb3350070df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"
Sep 17 14:24:54.784511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.784429    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"
Sep 17 14:24:54.815031 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:54.815002    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:55.069136 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:55.063865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"]
Sep 17 14:24:55.071069 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:24:55.071040    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f10f6b0_3830_4c87_a63e_2cb3350070df.slice/crio-55598473884fa906640513f06478b2f35a1547c3ff0c6d3a27f95f9130e72a9a WatchSource:0}: Error finding container 55598473884fa906640513f06478b2f35a1547c3ff0c6d3a27f95f9130e72a9a: Status 404 returned error can't find the container with id 55598473884fa906640513f06478b2f35a1547c3ff0c6d3a27f95f9130e72a9a
Sep 17 14:24:55.748059 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:55.747977    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:55.962229 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:55.962201    2565 generic.go:356] "Generic (PLEG): container finished" podID="5f10f6b0-3830-4c87-a63e-2cb3350070df" containerID="e76af364f6f580a1412b53713c47e8a3ae405f500a170084e239eda899d85714" exitCode=0
Sep 17 14:24:55.962393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:55.962292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl" event={"ID":"5f10f6b0-3830-4c87-a63e-2cb3350070df","Type":"ContainerDied","Data":"e76af364f6f580a1412b53713c47e8a3ae405f500a170084e239eda899d85714"}
Sep 17 14:24:55.962393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:55.962356    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl" event={"ID":"5f10f6b0-3830-4c87-a63e-2cb3350070df","Type":"ContainerStarted","Data":"55598473884fa906640513f06478b2f35a1547c3ff0c6d3a27f95f9130e72a9a"}
Sep 17 14:24:56.742521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:56.742486    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:57.114974 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:57.114923    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"
Sep 17 14:24:57.196109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:57.196073    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10f6b0-3830-4c87-a63e-2cb3350070df-maas-api-service-ca\") pod \"5f10f6b0-3830-4c87-a63e-2cb3350070df\" (UID: \"5f10f6b0-3830-4c87-a63e-2cb3350070df\") "
Sep 17 14:24:57.196544 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:57.196520    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f10f6b0-3830-4c87-a63e-2cb3350070df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5f10f6b0-3830-4c87-a63e-2cb3350070df" (UID: "5f10f6b0-3830-4c87-a63e-2cb3350070df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:24:57.196711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:57.196696    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f10f6b0-3830-4c87-a63e-2cb3350070df-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:24:57.734507 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:57.734475    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:57.974865 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:57.974837    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"
Sep 17 14:24:57.975284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:57.974872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl" event={"ID":"5f10f6b0-3830-4c87-a63e-2cb3350070df","Type":"ContainerDied","Data":"55598473884fa906640513f06478b2f35a1547c3ff0c6d3a27f95f9130e72a9a"}
Sep 17 14:24:57.975284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:57.974916    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55598473884fa906640513f06478b2f35a1547c3ff0c6d3a27f95f9130e72a9a"
Sep 17 14:24:58.699062 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:58.699031    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:58.826071 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:58.826039    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:24:59.677868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:24:59.677832    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:00.654293 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:00.654193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:05.522911 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:05.522880    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.507041 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.506997    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.574373 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.574339    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"]
Sep 17 14:25:06.574773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.574555    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz" podUID="1339c6df-9240-4fb6-89d6-f6820ab45d8c" containerName="payload-pre-processing" containerID="cri-o://41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888" gracePeriod=30
Sep 17 14:25:06.624715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.624677    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.729987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.729956    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.803141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.796434    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"]
Sep 17 14:25:06.803141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.801495    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2f37280d-vgrtl"]
Sep 17 14:25:06.817913 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.817875    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:06.853164 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.853133    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:25:06.905937 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.905907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1339c6df-9240-4fb6-89d6-f6820ab45d8c-plugins-config-volume\") pod \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\" (UID: \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\") "
Sep 17 14:25:06.906095 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.906076    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v7xt6\" (UniqueName: \"kubernetes.io/projected/1339c6df-9240-4fb6-89d6-f6820ab45d8c-kube-api-access-v7xt6\") pod \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\" (UID: \"1339c6df-9240-4fb6-89d6-f6820ab45d8c\") "
Sep 17 14:25:06.906411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.906384    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1339c6df-9240-4fb6-89d6-f6820ab45d8c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1339c6df-9240-4fb6-89d6-f6820ab45d8c" (UID: "1339c6df-9240-4fb6-89d6-f6820ab45d8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:06.906549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.906535    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1339c6df-9240-4fb6-89d6-f6820ab45d8c-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:06.908188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:06.908161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1339c6df-9240-4fb6-89d6-f6820ab45d8c-kube-api-access-v7xt6" (OuterVolumeSpecName: "kube-api-access-v7xt6") pod "1339c6df-9240-4fb6-89d6-f6820ab45d8c" (UID: "1339c6df-9240-4fb6-89d6-f6820ab45d8c"). InnerVolumeSpecName "kube-api-access-v7xt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:07.009136 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.009076    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7xt6\" (UniqueName: \"kubernetes.io/projected/1339c6df-9240-4fb6-89d6-f6820ab45d8c-kube-api-access-v7xt6\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:07.021760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.021726    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"]
Sep 17 14:25:07.022045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.021995    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy" containerID="cri-o://a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7" gracePeriod=30
Sep 17 14:25:07.028553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.028527    2565 generic.go:356] "Generic (PLEG): container finished" podID="1339c6df-9240-4fb6-89d6-f6820ab45d8c" containerID="41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888" exitCode=0
Sep 17 14:25:07.028654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.028566    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"
Sep 17 14:25:07.028654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.028604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz" event={"ID":"1339c6df-9240-4fb6-89d6-f6820ab45d8c","Type":"ContainerDied","Data":"41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888"}
Sep 17 14:25:07.028784 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.028658    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz" event={"ID":"1339c6df-9240-4fb6-89d6-f6820ab45d8c","Type":"ContainerDied","Data":"7e48b7a3308899ad811e762d92dcc64d0a6add8600bb27a02a6b6428f9c58def"}
Sep 17 14:25:07.028784 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.028693    2565 scope.go:117] "RemoveContainer" containerID="41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888"
Sep 17 14:25:07.039590 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.039571    2565 scope.go:117] "RemoveContainer" containerID="41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888"
Sep 17 14:25:07.039853 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:25:07.039833    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888\": container with ID starting with 41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888 not found: ID does not exist" containerID="41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888"
Sep 17 14:25:07.039924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.039875    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888"} err="failed to get container status \"41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888\": rpc error: code = NotFound desc = could not find container \"41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888\": container with ID starting with 41db11c4452bffdf6cb3dfc12249dc912ed267ba4ee4a2ca8e259002298a9888 not found: ID does not exist"
Sep 17 14:25:07.076360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.076327    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:07.170463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.170427    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"]
Sep 17 14:25:07.183099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.183072    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-2f37280d-6dd7d5cfbb-kbgqz"]
Sep 17 14:25:07.347242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.347155    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:07.519748 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.519713    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1339c6df-9240-4fb6-89d6-f6820ab45d8c" path="/var/lib/kubelet/pods/1339c6df-9240-4fb6-89d6-f6820ab45d8c/volumes"
Sep 17 14:25:07.520076 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.520062    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f10f6b0-3830-4c87-a63e-2cb3350070df" path="/var/lib/kubelet/pods/5f10f6b0-3830-4c87-a63e-2cb3350070df/volumes"
Sep 17 14:25:07.728165 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.728080    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:07.826571 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.826542    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:07.865181 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.865151    2565 patch_prober.go:28] interesting pod/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Sep 17 14:25:07.865341 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:07.865217    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:25:08.218544 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:08.218501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:08.359389 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:08.359345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:08.488603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:08.488521    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"]
Sep 17 14:25:08.488775 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:08.488752    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy" containerID="cri-o://3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64" gracePeriod=30
Sep 17 14:25:08.526434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:08.526395    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:08.981467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:08.981435    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:09.300089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.300018    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"]
Sep 17 14:25:09.300478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.300463    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f10f6b0-3830-4c87-a63e-2cb3350070df" containerName="revoke-keys"
Sep 17 14:25:09.300535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.300479    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f10f6b0-3830-4c87-a63e-2cb3350070df" containerName="revoke-keys"
Sep 17 14:25:09.300535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.300492    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1339c6df-9240-4fb6-89d6-f6820ab45d8c" containerName="payload-pre-processing"
Sep 17 14:25:09.300535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.300498    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1339c6df-9240-4fb6-89d6-f6820ab45d8c" containerName="payload-pre-processing"
Sep 17 14:25:09.300667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.300570    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1339c6df-9240-4fb6-89d6-f6820ab45d8c" containerName="payload-pre-processing"
Sep 17 14:25:09.300667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.300580    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f10f6b0-3830-4c87-a63e-2cb3350070df" containerName="revoke-keys"
Sep 17 14:25:09.303531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.303512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.310164 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.310120    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-0e6aaa-openshift-default-dockercfg-kdmqd\""
Sep 17 14:25:09.323656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.323627    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"]
Sep 17 14:25:09.328813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.328785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-socket\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.328926 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.328890    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istiod-ca-cert\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.328980 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.328961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-data\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.329079 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.329059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-certs\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.329126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.329114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-credential-socket\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.329211 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.329188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-podinfo\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.329296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.329277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-envoy\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.329361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.329316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-token\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.329361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.329347    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cjzj\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-kube-api-access-6cjzj\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.338999 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.338976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430138    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-socket\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istiod-ca-cert\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-data\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-certs\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-credential-socket\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-podinfo\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-envoy\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-token\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cjzj\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-kube-api-access-6cjzj\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.430731    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-socket\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.431050    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-certs\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.431100    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-data\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.431615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.431555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-credential-socket\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.432536 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.432209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istiod-ca-cert\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.435801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.434944    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-podinfo\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.436459 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.436427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-envoy\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.442224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.442175    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-token\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.442384 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.442345    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cjzj\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-kube-api-access-6cjzj\") pod \"e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") " pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.442995 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.442969    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:09.602641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.602602    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:09.613301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.613245    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:09.907555 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:25:09.907525    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeec9f8e_d138_4d04_8339_efbf73e5ff2e.slice/crio-9a2b0ca8215938e7bbc611ad1368b2b6d781e376ef0ac51af9a08476fbac278e WatchSource:0}: Error finding container 9a2b0ca8215938e7bbc611ad1368b2b6d781e376ef0ac51af9a08476fbac278e: Status 404 returned error can't find the container with id 9a2b0ca8215938e7bbc611ad1368b2b6d781e376ef0ac51af9a08476fbac278e
Sep 17 14:25:09.907699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.907678    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"]
Sep 17 14:25:09.909957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.909927    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:25:09.910046 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.909989    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:25:09.910046 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:09.910017    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:25:10.049321 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:10.048958    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" event={"ID":"feec9f8e-d138-4d04-8339-efbf73e5ff2e","Type":"ContainerStarted","Data":"7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b"}
Sep 17 14:25:10.049321 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:10.049028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" event={"ID":"feec9f8e-d138-4d04-8339-efbf73e5ff2e","Type":"ContainerStarted","Data":"9a2b0ca8215938e7bbc611ad1368b2b6d781e376ef0ac51af9a08476fbac278e"}
Sep 17 14:25:10.086382 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:10.086331    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" podStartSLOduration=1.086313127 podStartE2EDuration="1.086313127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:25:10.083777453 +0000 UTC m=+1097.017282645" watchObservedRunningTime="2026-09-17 14:25:10.086313127 +0000 UTC m=+1097.019818319"
Sep 17 14:25:10.200342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:10.200234    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:10.614475 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:10.614437    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:10.619680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:10.619652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:11.054506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:11.054427    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:11.055468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:11.055449    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:11.180106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:11.180072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:12.138268 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.138236    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:12.277441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.277415    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:25:12.290073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.290015    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:12.363424 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363394    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34058834-c753-4e81-8391-066fcd936639-istiod-ca-cert\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363424 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363431    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-socket\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363468    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-certs\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363515    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34058834-c753-4e81-8391-066fcd936639-istio-podinfo\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363564    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-istio-token\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363599    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-credential-socket\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363650    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-data\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363682    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n684p\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-kube-api-access-n684p\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363700    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.363923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363770    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-envoy\") pod \"34058834-c753-4e81-8391-066fcd936639\" (UID: \"34058834-c753-4e81-8391-066fcd936639\") "
Sep 17 14:25:12.363923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363830    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.363923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363869    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-data" (OuterVolumeSpecName: "istio-data") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.363923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.363905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.364199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.364026    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34058834-c753-4e81-8391-066fcd936639-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:12.364288 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.364205    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.364288 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.364236    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.364288 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.364279    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34058834-c753-4e81-8391-066fcd936639-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.364463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.364293    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.364463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.364305    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.365867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.365837    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-kube-api-access-n684p" (OuterVolumeSpecName: "kube-api-access-n684p") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "kube-api-access-n684p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:12.365867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.365856    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-istio-token" (OuterVolumeSpecName: "istio-token") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:12.366038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.365904    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:12.366038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.365927    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34058834-c753-4e81-8391-066fcd936639-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34058834-c753-4e81-8391-066fcd936639" (UID: "34058834-c753-4e81-8391-066fcd936639"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:25:12.465739 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.465702    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34058834-c753-4e81-8391-066fcd936639-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.465739 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.465733    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.465739 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.465746    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n684p\" (UniqueName: \"kubernetes.io/projected/34058834-c753-4e81-8391-066fcd936639-kube-api-access-n684p\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.465968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.465760    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34058834-c753-4e81-8391-066fcd936639-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:12.472304 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:12.472271    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:13.068206 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.068173    2565 generic.go:356] "Generic (PLEG): container finished" podID="34058834-c753-4e81-8391-066fcd936639" containerID="a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7" exitCode=0
Sep 17 14:25:13.068466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.068227    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"
Sep 17 14:25:13.068466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.068248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" event={"ID":"34058834-c753-4e81-8391-066fcd936639","Type":"ContainerDied","Data":"a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7"}
Sep 17 14:25:13.068466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.068317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c" event={"ID":"34058834-c753-4e81-8391-066fcd936639","Type":"ContainerDied","Data":"a36d148407ab46fd3b6cf10744ab784928fe0b250589d7d4abae09b51d5dc979"}
Sep 17 14:25:13.068466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.068339    2565 scope.go:117] "RemoveContainer" containerID="a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7"
Sep 17 14:25:13.080856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.080838    2565 scope.go:117] "RemoveContainer" containerID="e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1"
Sep 17 14:25:13.089174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.089157    2565 scope.go:117] "RemoveContainer" containerID="a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7"
Sep 17 14:25:13.089450 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:25:13.089431    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7\": container with ID starting with a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7 not found: ID does not exist" containerID="a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7"
Sep 17 14:25:13.089503 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.089458    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7"} err="failed to get container status \"a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7\": rpc error: code = NotFound desc = could not find container \"a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7\": container with ID starting with a6d4f5cd177d47432114e81a730411f3d31daac7fe35c5b26222c1f1ddda0fe7 not found: ID does not exist"
Sep 17 14:25:13.089503 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.089474    2565 scope.go:117] "RemoveContainer" containerID="e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1"
Sep 17 14:25:13.089726 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:25:13.089708    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1\": container with ID starting with e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1 not found: ID does not exist" containerID="e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1"
Sep 17 14:25:13.089768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.089732    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1"} err="failed to get container status \"e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1\": rpc error: code = NotFound desc = could not find container \"e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1\": container with ID starting with e6cb8cfaaf9cb9d3eca2c22d3f650e7537776b1c32aad4236d7aa3282ff341e1 not found: ID does not exist"
Sep 17 14:25:13.094500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.094476    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"]
Sep 17 14:25:13.100618 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.100596    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2f37280d-openshift-default-647575f4d-btg4c"]
Sep 17 14:25:13.458602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.458572    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:13.520967 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.520939    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34058834-c753-4e81-8391-066fcd936639" path="/var/lib/kubelet/pods/34058834-c753-4e81-8391-066fcd936639/volumes"
Sep 17 14:25:13.766650 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.766619    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:25:13.883573 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883523    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-credential-socket\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.883768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883597    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istiod-ca-cert\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.883768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883619    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-certs\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.883768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883634    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-envoy\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.883768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883680    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-podinfo\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.883768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883712    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-socket\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.883768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-token\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.884108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883830    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-data\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.884108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883828    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:13.884108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883897    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfsbg\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-kube-api-access-zfsbg\") pod \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\" (UID: \"7f0ec2bc-d7f5-4743-9558-065b0336f86b\") "
Sep 17 14:25:13.884108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.883960    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:13.884108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.884054    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:25:13.884108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.884054    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:13.884440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.884164    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:13.884440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.884377    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:13.884440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.884397    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:13.884440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.884411    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:13.884440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.884424    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:13.884719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.884444    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:13.885941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.885910    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:25:13.886054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.885956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:13.886054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.886020    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:25:13.886054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.886029    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-kube-api-access-zfsbg" (OuterVolumeSpecName: "kube-api-access-zfsbg") pod "7f0ec2bc-d7f5-4743-9558-065b0336f86b" (UID: "7f0ec2bc-d7f5-4743-9558-065b0336f86b"). InnerVolumeSpecName "kube-api-access-zfsbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:25:13.985604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.985573    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsbg\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-kube-api-access-zfsbg\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:13.985604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.985602    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:13.985777 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.985612    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:13.985777 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:13.985621    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f0ec2bc-d7f5-4743-9558-065b0336f86b-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:25:14.076898 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.076869    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerID="3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64" exitCode=0
Sep 17 14:25:14.077032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.076906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" event={"ID":"7f0ec2bc-d7f5-4743-9558-065b0336f86b","Type":"ContainerDied","Data":"3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64"}
Sep 17 14:25:14.077032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.076966    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk" event={"ID":"7f0ec2bc-d7f5-4743-9558-065b0336f86b","Type":"ContainerDied","Data":"808d269397db4590a5e4355fc67ffa1f335877c8062ebb74362ae26494afbcf6"}
Sep 17 14:25:14.077032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.076989    2565 scope.go:117] "RemoveContainer" containerID="3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64"
Sep 17 14:25:14.077032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.076909    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"
Sep 17 14:25:14.087206 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.087191    2565 scope.go:117] "RemoveContainer" containerID="a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b"
Sep 17 14:25:14.095316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.095297    2565 scope.go:117] "RemoveContainer" containerID="3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64"
Sep 17 14:25:14.095572 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:25:14.095552    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64\": container with ID starting with 3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64 not found: ID does not exist" containerID="3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64"
Sep 17 14:25:14.095622 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.095579    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64"} err="failed to get container status \"3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64\": rpc error: code = NotFound desc = could not find container \"3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64\": container with ID starting with 3573d68cd21d0f4e301830a0839b496e4ee13454cd20792c9ea54f2e8dd1bc64 not found: ID does not exist"
Sep 17 14:25:14.095622 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.095596    2565 scope.go:117] "RemoveContainer" containerID="a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b"
Sep 17 14:25:14.095840 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:25:14.095822    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b\": container with ID starting with a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b not found: ID does not exist" containerID="a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b"
Sep 17 14:25:14.095896 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.095849    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b"} err="failed to get container status \"a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b\": rpc error: code = NotFound desc = could not find container \"a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b\": container with ID starting with a2b8a1c300a0b44c7be3ca4e6a1c20f81dcfd8a1b8bd93cf5ee72b4b5527705b not found: ID does not exist"
Sep 17 14:25:14.105124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.105101    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"]
Sep 17 14:25:14.109477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.109460    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-86c464-openshift-default-5d966fdd44-qlqvk"]
Sep 17 14:25:14.185412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:14.185377    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:15.244330 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:15.244293    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:15.519552 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:15.519473    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" path="/var/lib/kubelet/pods/7f0ec2bc-d7f5-4743-9558-065b0336f86b/volumes"
Sep 17 14:25:16.279383 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:16.279346    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:17.471232 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:17.471200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:17.789326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:17.789203    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:18.031384 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.031344    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:18.413946 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.413905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:18.540856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.540824    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"]
Sep 17 14:25:18.541221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541208    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy"
Sep 17 14:25:18.541275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541222    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy"
Sep 17 14:25:18.541275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541238    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy"
Sep 17 14:25:18.541275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541244    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy"
Sep 17 14:25:18.541275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541274    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy"
Sep 17 14:25:18.541406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541280    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy"
Sep 17 14:25:18.541406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541341    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy"
Sep 17 14:25:18.541406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541349    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy"
Sep 17 14:25:18.541406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.541356    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="34058834-c753-4e81-8391-066fcd936639" containerName="istio-proxy"
Sep 17 14:25:18.544374 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.544353    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:18.547325 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.547284    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:18.547490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.547343    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-0e6aaa-dockercfg-4656d\""
Sep 17 14:25:18.549407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.549383    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-0e6aaa\""
Sep 17 14:25:18.555701 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.555677    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"]
Sep 17 14:25:18.640279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.635034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a5382a-52ce-4b24-b87e-631bd10b8533-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7\" (UID: \"69a5382a-52ce-4b24-b87e-631bd10b8533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:18.640279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.635172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqjzb\" (UniqueName: \"kubernetes.io/projected/69a5382a-52ce-4b24-b87e-631bd10b8533-kube-api-access-nqjzb\") pod \"payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7\" (UID: \"69a5382a-52ce-4b24-b87e-631bd10b8533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:18.645467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.645435    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"]
Sep 17 14:25:18.646423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.646396    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy"
Sep 17 14:25:18.646588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.646575    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy"
Sep 17 14:25:18.646778 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.646764    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f0ec2bc-d7f5-4743-9558-065b0336f86b" containerName="istio-proxy"
Sep 17 14:25:18.651753 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.651733    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:18.674107 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.674065    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"]
Sep 17 14:25:18.736662 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.736624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqjzb\" (UniqueName: \"kubernetes.io/projected/69a5382a-52ce-4b24-b87e-631bd10b8533-kube-api-access-nqjzb\") pod \"payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7\" (UID: \"69a5382a-52ce-4b24-b87e-631bd10b8533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:18.736830 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.736700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434e7380-24db-42cd-b24b-15dff2c00e6d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h\" (UID: \"434e7380-24db-42cd-b24b-15dff2c00e6d\") " pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:18.736830 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.736731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxcs2\" (UniqueName: \"kubernetes.io/projected/434e7380-24db-42cd-b24b-15dff2c00e6d-kube-api-access-pxcs2\") pod \"payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h\" (UID: \"434e7380-24db-42cd-b24b-15dff2c00e6d\") " pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:18.736830 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.736822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a5382a-52ce-4b24-b87e-631bd10b8533-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7\" (UID: \"69a5382a-52ce-4b24-b87e-631bd10b8533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:18.737560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.737538    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a5382a-52ce-4b24-b87e-631bd10b8533-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7\" (UID: \"69a5382a-52ce-4b24-b87e-631bd10b8533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:18.745952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.745927    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqjzb\" (UniqueName: \"kubernetes.io/projected/69a5382a-52ce-4b24-b87e-631bd10b8533-kube-api-access-nqjzb\") pod \"payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7\" (UID: \"69a5382a-52ce-4b24-b87e-631bd10b8533\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:18.838360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.838322    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434e7380-24db-42cd-b24b-15dff2c00e6d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h\" (UID: \"434e7380-24db-42cd-b24b-15dff2c00e6d\") " pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:18.838571 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.838380    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxcs2\" (UniqueName: \"kubernetes.io/projected/434e7380-24db-42cd-b24b-15dff2c00e6d-kube-api-access-pxcs2\") pod \"payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h\" (UID: \"434e7380-24db-42cd-b24b-15dff2c00e6d\") " pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:18.839085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.839061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434e7380-24db-42cd-b24b-15dff2c00e6d-plugins-config-volume\") pod \"payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h\" (UID: \"434e7380-24db-42cd-b24b-15dff2c00e6d\") " pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:18.848272 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.848231    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxcs2\" (UniqueName: \"kubernetes.io/projected/434e7380-24db-42cd-b24b-15dff2c00e6d-kube-api-access-pxcs2\") pod \"payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h\" (UID: \"434e7380-24db-42cd-b24b-15dff2c00e6d\") " pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:18.857014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.856987    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:18.907362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.907146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:18.975027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:18.974941    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:19.054792 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:19.054707    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:19.138193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:19.138160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"]
Sep 17 14:25:19.138940 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:25:19.138913    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a5382a_52ce_4b24_b87e_631bd10b8533.slice/crio-ccfce008fcc343735ba8c19a0c01fe27259cbf9ee4d80fa6ee861c22b53686fe WatchSource:0}: Error finding container ccfce008fcc343735ba8c19a0c01fe27259cbf9ee4d80fa6ee861c22b53686fe: Status 404 returned error can't find the container with id ccfce008fcc343735ba8c19a0c01fe27259cbf9ee4d80fa6ee861c22b53686fe
Sep 17 14:25:19.172734 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:19.172703    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:19.264517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:19.264492    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"]
Sep 17 14:25:19.267587 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:25:19.267555    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434e7380_24db_42cd_b24b_15dff2c00e6d.slice/crio-f0cea423c1f6a433da56e13b215c01906082788dcf2e407b4075ebbed40bcc1f WatchSource:0}: Error finding container f0cea423c1f6a433da56e13b215c01906082788dcf2e407b4075ebbed40bcc1f: Status 404 returned error can't find the container with id f0cea423c1f6a433da56e13b215c01906082788dcf2e407b4075ebbed40bcc1f
Sep 17 14:25:19.304844 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:19.304821    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:19.685781 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:19.685725    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:20.113646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.113611    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7" event={"ID":"69a5382a-52ce-4b24-b87e-631bd10b8533","Type":"ContainerStarted","Data":"382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683"}
Sep 17 14:25:20.113824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.113684    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:20.113824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.113710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7" event={"ID":"69a5382a-52ce-4b24-b87e-631bd10b8533","Type":"ContainerStarted","Data":"ccfce008fcc343735ba8c19a0c01fe27259cbf9ee4d80fa6ee861c22b53686fe"}
Sep 17 14:25:20.115317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.115289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h" event={"ID":"434e7380-24db-42cd-b24b-15dff2c00e6d","Type":"ContainerStarted","Data":"e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96"}
Sep 17 14:25:20.115429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.115337    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h" event={"ID":"434e7380-24db-42cd-b24b-15dff2c00e6d","Type":"ContainerStarted","Data":"f0cea423c1f6a433da56e13b215c01906082788dcf2e407b4075ebbed40bcc1f"}
Sep 17 14:25:20.115429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.115372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:20.140566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.140519    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7" podStartSLOduration=2.14049987 podStartE2EDuration="2.14049987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:25:20.136864779 +0000 UTC m=+1107.070369972" watchObservedRunningTime="2026-09-17 14:25:20.14049987 +0000 UTC m=+1107.074005065"
Sep 17 14:25:20.159973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.159865    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h" podStartSLOduration=2.159846056 podStartE2EDuration="2.159846056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:25:20.159780404 +0000 UTC m=+1107.093285597" watchObservedRunningTime="2026-09-17 14:25:20.159846056 +0000 UTC m=+1107.093351249"
Sep 17 14:25:20.475291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:20.475201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:31.124417 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:31.124379    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:25:31.124877 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:31.124557    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:25:34.604946 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:34.604910    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:36.898924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:36.898889    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:38.232638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:38.232603    2565 generic.go:356] "Generic (PLEG): container finished" podID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerID="7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b" exitCode=0
Sep 17 14:25:38.233184 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:38.232681    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" event={"ID":"feec9f8e-d138-4d04-8339-efbf73e5ff2e","Type":"ContainerDied","Data":"7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b"}
Sep 17 14:25:38.233299 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:38.233281    2565 scope.go:117] "RemoveContainer" containerID="7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b"
Sep 17 14:25:38.234499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:38.234468    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:25:38.234600 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:38.234537    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:25:38.234600 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:38.234569    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:25:39.240930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:39.240887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" event={"ID":"feec9f8e-d138-4d04-8339-efbf73e5ff2e","Type":"ContainerStarted","Data":"61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7"}
Sep 17 14:25:39.614403 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:39.614372    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:39.615279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:39.615228    2565 patch_prober.go:28] interesting pod/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 17 14:25:39.615408 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:39.615316    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 17 14:25:40.614182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:40.614142    2565 patch_prober.go:28] interesting pod/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 17 14:25:40.614703 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:40.614214    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 17 14:25:41.054737 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:41.054636    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:41.343030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:41.342976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:41.500583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:41.500549    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:41.614319 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:41.614213    2565 patch_prober.go:28] interesting pod/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 17 14:25:41.614319 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:41.614286    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 17 14:25:42.174781 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:42.174743    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:42.563926 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:42.563823    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:42.614285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:42.614220    2565 patch_prober.go:28] interesting pod/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 17 14:25:42.614617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:42.614580    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 17 14:25:43.320570 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:43.320533    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:43.613746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:43.613706    2565 patch_prober.go:28] interesting pod/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Sep 17 14:25:43.613955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:43.613770    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Sep 17 14:25:44.526376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:44.526340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:44.618641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:44.618607    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:44.619813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:44.619789    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:25:45.730838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:45.730801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:45.913627 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:45.913589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:46.946624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:46.946592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:54.189462 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:54.189427    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:54.458155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:54.458074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:54.647724 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:54.647695    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:25:55.894377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:25:55.894345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:10.373367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:10.373326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:16.934500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:16.934407    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:17.402086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:17.402048    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:24.476517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.476484    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"]
Sep 17 14:26:24.479029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.479012    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.483741 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.483721    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0e6aaa\"/\"kube-root-ca.crt\""
Sep 17 14:26:24.483741 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.483740    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0e6aaa\"/\"default-dockercfg-bgnrb\""
Sep 17 14:26:24.483936 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.483781    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-0e6aaa\"/\"openshift-service-ca.crt\""
Sep 17 14:26:24.483936 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.483845    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-0e6aaa\"/\"explicit-model-0e6aaa-kserve-self-signed-certs\""
Sep 17 14:26:24.501006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.500982    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"]
Sep 17 14:26:24.531863 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.531842    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:24.590090 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.590049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-tmp-dir\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.590225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.590121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq5lj\" (UniqueName: \"kubernetes.io/projected/b1a93061-14ee-462f-8421-7abd69bbad6d-kube-api-access-cq5lj\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.590225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.590159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-home\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.590366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.590225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-dshm\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.590366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.590358    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-model-cache\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.590457 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.590420    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a93061-14ee-462f-8421-7abd69bbad6d-tls-certs\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.691840 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.691809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq5lj\" (UniqueName: \"kubernetes.io/projected/b1a93061-14ee-462f-8421-7abd69bbad6d-kube-api-access-cq5lj\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.692036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.691857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-home\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.692036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.691879    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-dshm\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.692036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.691924    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-model-cache\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.692036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.691974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a93061-14ee-462f-8421-7abd69bbad6d-tls-certs\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.692306 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.692055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-tmp-dir\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.692306 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.692290    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-home\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.692429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.692366    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-model-cache\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.692429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.692406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-tmp-dir\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.694140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.694121    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-dshm\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.694442 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.694426    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a93061-14ee-462f-8421-7abd69bbad6d-tls-certs\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.703349 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.703323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq5lj\" (UniqueName: \"kubernetes.io/projected/b1a93061-14ee-462f-8421-7abd69bbad6d-kube-api-access-cq5lj\") pod \"explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") " pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:24.789622 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:24.789538    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:25.076594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:25.076570    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"]
Sep 17 14:26:25.078985 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:26:25.078955    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1a93061_14ee_462f_8421_7abd69bbad6d.slice/crio-373e35372837647f26073cbbbdc8bbed2a587e950a87dcc0c817bc5ec81ecb76 WatchSource:0}: Error finding container 373e35372837647f26073cbbbdc8bbed2a587e950a87dcc0c817bc5ec81ecb76: Status 404 returned error can't find the container with id 373e35372837647f26073cbbbdc8bbed2a587e950a87dcc0c817bc5ec81ecb76
Sep 17 14:26:25.531129 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:25.531042    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5" event={"ID":"b1a93061-14ee-462f-8421-7abd69bbad6d","Type":"ContainerStarted","Data":"0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1"}
Sep 17 14:26:25.531129 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:25.531088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5" event={"ID":"b1a93061-14ee-462f-8421-7abd69bbad6d","Type":"ContainerStarted","Data":"373e35372837647f26073cbbbdc8bbed2a587e950a87dcc0c817bc5ec81ecb76"}
Sep 17 14:26:25.570571 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:25.570520    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5" podStartSLOduration=1.570506053 podStartE2EDuration="1.570506053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:26:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:26:25.570418041 +0000 UTC m=+1172.503923236" watchObservedRunningTime="2026-09-17 14:26:25.570506053 +0000 UTC m=+1172.504011244"
Sep 17 14:26:27.775487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:27.775453    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:28.541859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:28.541829    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:34.790080 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:34.790041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:34.790670 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:34.790095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:34.803305 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:34.803280    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:35.488444 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:35.488405    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:35.604916 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:35.604877    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:26:35.665712 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:35.665662    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:35.841508 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:35.841479    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:36.505801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:36.505763    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:36.705487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:36.705440    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:38.507708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:38.507675    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:47.918745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:47.918712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:48.071379 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:48.071349    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:48.705267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:48.705232    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:53.341911 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.341875    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"]
Sep 17 14:26:53.380732 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.380689    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:53.474362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.474329    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"]
Sep 17 14:26:53.477845 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.477826    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"
Sep 17 14:26:53.481362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.481334    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:26:53.481520 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.481437    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:26:53.538106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.538073    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:26:53.540648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.540620    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:26:53.543912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.543892    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:26:53.545996 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.545973    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:26:53.554728 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.554699    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"]
Sep 17 14:26:53.611661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.611577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2feffe-60d6-4208-b8f5-21a74ee449e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn\" (UID: \"5d2feffe-60d6-4208-b8f5-21a74ee449e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"
Sep 17 14:26:53.712268 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.712231    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2feffe-60d6-4208-b8f5-21a74ee449e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn\" (UID: \"5d2feffe-60d6-4208-b8f5-21a74ee449e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"
Sep 17 14:26:53.712835 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.712814    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2feffe-60d6-4208-b8f5-21a74ee449e1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn\" (UID: \"5d2feffe-60d6-4208-b8f5-21a74ee449e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"
Sep 17 14:26:53.789546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:53.789517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"
Sep 17 14:26:54.092973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:54.092944    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"]
Sep 17 14:26:54.095413 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:26:54.095384    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d2feffe_60d6_4208_b8f5_21a74ee449e1.slice/crio-1d008c9e82ac8b33baccb6fe33b4a1816f675cdb386328b7e76e04541cbdaf07 WatchSource:0}: Error finding container 1d008c9e82ac8b33baccb6fe33b4a1816f675cdb386328b7e76e04541cbdaf07: Status 404 returned error can't find the container with id 1d008c9e82ac8b33baccb6fe33b4a1816f675cdb386328b7e76e04541cbdaf07
Sep 17 14:26:54.662214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:54.662137    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:54.712159 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:54.712125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn" event={"ID":"5d2feffe-60d6-4208-b8f5-21a74ee449e1","Type":"ContainerStarted","Data":"5872b646059ed798135ea222e02216f5ab7e124c582434189ec826bbb2b8fcf2"}
Sep 17 14:26:54.712348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:54.712185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn" event={"ID":"5d2feffe-60d6-4208-b8f5-21a74ee449e1","Type":"ContainerStarted","Data":"1d008c9e82ac8b33baccb6fe33b4a1816f675cdb386328b7e76e04541cbdaf07"}
Sep 17 14:26:54.746987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:54.746931    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn" podStartSLOduration=1.746915752 podStartE2EDuration="1.746915752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:26:54.745160125 +0000 UTC m=+1201.678665318" watchObservedRunningTime="2026-09-17 14:26:54.746915752 +0000 UTC m=+1201.680420943"
Sep 17 14:26:55.719962 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:55.719924    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d2feffe-60d6-4208-b8f5-21a74ee449e1" containerID="5872b646059ed798135ea222e02216f5ab7e124c582434189ec826bbb2b8fcf2" exitCode=0
Sep 17 14:26:55.720393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:55.719980    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn" event={"ID":"5d2feffe-60d6-4208-b8f5-21a74ee449e1","Type":"ContainerDied","Data":"5872b646059ed798135ea222e02216f5ab7e124c582434189ec826bbb2b8fcf2"}
Sep 17 14:26:55.970620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:55.970527    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:56.874028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:56.874007    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"
Sep 17 14:26:56.945833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:56.945802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2feffe-60d6-4208-b8f5-21a74ee449e1-maas-api-service-ca\") pod \"5d2feffe-60d6-4208-b8f5-21a74ee449e1\" (UID: \"5d2feffe-60d6-4208-b8f5-21a74ee449e1\") "
Sep 17 14:26:56.946154 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:56.946131    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d2feffe-60d6-4208-b8f5-21a74ee449e1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5d2feffe-60d6-4208-b8f5-21a74ee449e1" (UID: "5d2feffe-60d6-4208-b8f5-21a74ee449e1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:26:57.047003 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:57.046918    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d2feffe-60d6-4208-b8f5-21a74ee449e1-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:26:57.514367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:57.514330    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:57.733614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:57.733584    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"
Sep 17 14:26:57.733789 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:57.733578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn" event={"ID":"5d2feffe-60d6-4208-b8f5-21a74ee449e1","Type":"ContainerDied","Data":"1d008c9e82ac8b33baccb6fe33b4a1816f675cdb386328b7e76e04541cbdaf07"}
Sep 17 14:26:57.733789 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:57.733706    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d008c9e82ac8b33baccb6fe33b4a1816f675cdb386328b7e76e04541cbdaf07"
Sep 17 14:26:58.982560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:58.982525    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:59.552603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:59.552566    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:26:59.737544 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:26:59.737516    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:00.023227 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.023199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:00.304137 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.304062    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"]
Sep 17 14:27:00.304563 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.304550    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d2feffe-60d6-4208-b8f5-21a74ee449e1" containerName="revoke-keys"
Sep 17 14:27:00.304616 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.304565    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d2feffe-60d6-4208-b8f5-21a74ee449e1" containerName="revoke-keys"
Sep 17 14:27:00.304669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.304657    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d2feffe-60d6-4208-b8f5-21a74ee449e1" containerName="revoke-keys"
Sep 17 14:27:00.307864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.307848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.318818 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.318796    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-130b7aa1-openshift-default-dockercfg-hkww2\""
Sep 17 14:27:00.335165 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.335139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"]
Sep 17 14:27:00.382758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.382736    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-istio-token\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.382883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.382770    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-socket\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.382883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.382793    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-data\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.382883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.382836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cdd559-2db2-46a1-8afa-81da80143208-istiod-ca-cert\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.382883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.382875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-certs\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.383025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.382958    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-envoy\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.383025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.383009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpqbf\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-kube-api-access-wpqbf\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.383091 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.383040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cdd559-2db2-46a1-8afa-81da80143208-istio-podinfo\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.383091 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.383075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-credential-socket\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.384533 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.384514    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:00.484231 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpqbf\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-kube-api-access-wpqbf\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cdd559-2db2-46a1-8afa-81da80143208-istio-podinfo\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484301    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-credential-socket\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-istio-token\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484466    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-socket\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-data\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cdd559-2db2-46a1-8afa-81da80143208-istiod-ca-cert\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-certs\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484818 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484671    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-envoy\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484818 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484706    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-credential-socket\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484818 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484796    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-socket\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-data\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.484987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.484966    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-certs\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.485201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.485181    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cdd559-2db2-46a1-8afa-81da80143208-istiod-ca-cert\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.486759 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.486742    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-envoy\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.486890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.486874    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cdd559-2db2-46a1-8afa-81da80143208-istio-podinfo\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.507122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.507095    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpqbf\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-kube-api-access-wpqbf\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.515607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.515581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-istio-token\") pod \"e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") " pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.613773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.613744    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:00.621654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.618930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:00.757328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.747446    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:00.917013 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.916980    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"]
Sep 17 14:27:00.918323 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:27:00.918298    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cdd559_2db2_46a1_8afa_81da80143208.slice/crio-d83e349a4726b7b8905bb07396d4236712114c24dc63980b31580f8d13695024 WatchSource:0}: Error finding container d83e349a4726b7b8905bb07396d4236712114c24dc63980b31580f8d13695024: Status 404 returned error can't find the container with id d83e349a4726b7b8905bb07396d4236712114c24dc63980b31580f8d13695024
Sep 17 14:27:00.920630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.920599    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:00.920694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.920661    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:00.920694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:00.920690    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:01.768983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:01.768942    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" event={"ID":"c1cdd559-2db2-46a1-8afa-81da80143208","Type":"ContainerStarted","Data":"7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694"}
Sep 17 14:27:01.769381 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:01.769006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" event={"ID":"c1cdd559-2db2-46a1-8afa-81da80143208","Type":"ContainerStarted","Data":"d83e349a4726b7b8905bb07396d4236712114c24dc63980b31580f8d13695024"}
Sep 17 14:27:01.817533 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:01.817488    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" podStartSLOduration=1.817471491 podStartE2EDuration="1.817471491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:27:01.816064146 +0000 UTC m=+1208.749569337" watchObservedRunningTime="2026-09-17 14:27:01.817471491 +0000 UTC m=+1208.750976682"
Sep 17 14:27:02.185820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:02.185784    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:02.619909 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:02.619871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:02.625142 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:02.625117    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:02.775887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:02.775856    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:02.776899 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:02.776875    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:03.634985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:03.634939    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:04.272663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:04.272633    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:04.625826 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:04.625792    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:05.634437 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:05.634401    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:06.965543 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:06.965509    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:07.916365 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:07.916329    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:08.363115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.363075    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5" podUID="b1a93061-14ee-462f-8421-7abd69bbad6d" containerName="main" containerID="cri-o://0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1" gracePeriod=45
Sep 17 14:27:08.612449 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.612424    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:27:08.683680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.683580    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-model-cache\") pod \"b1a93061-14ee-462f-8421-7abd69bbad6d\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") "
Sep 17 14:27:08.683859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.683713    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq5lj\" (UniqueName: \"kubernetes.io/projected/b1a93061-14ee-462f-8421-7abd69bbad6d-kube-api-access-cq5lj\") pod \"b1a93061-14ee-462f-8421-7abd69bbad6d\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") "
Sep 17 14:27:08.683859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.683760    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-home\") pod \"b1a93061-14ee-462f-8421-7abd69bbad6d\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") "
Sep 17 14:27:08.683859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.683780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-dshm\") pod \"b1a93061-14ee-462f-8421-7abd69bbad6d\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") "
Sep 17 14:27:08.684045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.683854    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1a93061-14ee-462f-8421-7abd69bbad6d" (UID: "b1a93061-14ee-462f-8421-7abd69bbad6d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:08.684045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.683862    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-tmp-dir\") pod \"b1a93061-14ee-462f-8421-7abd69bbad6d\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") "
Sep 17 14:27:08.684045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.683919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a93061-14ee-462f-8421-7abd69bbad6d-tls-certs\") pod \"b1a93061-14ee-462f-8421-7abd69bbad6d\" (UID: \"b1a93061-14ee-462f-8421-7abd69bbad6d\") "
Sep 17 14:27:08.684045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.684035    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-home" (OuterVolumeSpecName: "home") pod "b1a93061-14ee-462f-8421-7abd69bbad6d" (UID: "b1a93061-14ee-462f-8421-7abd69bbad6d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:08.684282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.684052    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1a93061-14ee-462f-8421-7abd69bbad6d" (UID: "b1a93061-14ee-462f-8421-7abd69bbad6d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:08.684426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.684404    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:08.684489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.684428    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:08.684489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.684438    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:08.686271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.686232    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-dshm" (OuterVolumeSpecName: "dshm") pod "b1a93061-14ee-462f-8421-7abd69bbad6d" (UID: "b1a93061-14ee-462f-8421-7abd69bbad6d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:08.686471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.686435    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1a93061-14ee-462f-8421-7abd69bbad6d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1a93061-14ee-462f-8421-7abd69bbad6d" (UID: "b1a93061-14ee-462f-8421-7abd69bbad6d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:27:08.686562 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.686469    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1a93061-14ee-462f-8421-7abd69bbad6d-kube-api-access-cq5lj" (OuterVolumeSpecName: "kube-api-access-cq5lj") pod "b1a93061-14ee-462f-8421-7abd69bbad6d" (UID: "b1a93061-14ee-462f-8421-7abd69bbad6d"). InnerVolumeSpecName "kube-api-access-cq5lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:08.785317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.785284    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1a93061-14ee-462f-8421-7abd69bbad6d-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:08.785317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.785318    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1a93061-14ee-462f-8421-7abd69bbad6d-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:08.785494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.785335    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq5lj\" (UniqueName: \"kubernetes.io/projected/b1a93061-14ee-462f-8421-7abd69bbad6d-kube-api-access-cq5lj\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:08.811801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.811767    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1a93061-14ee-462f-8421-7abd69bbad6d" containerID="0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1" exitCode=0
Sep 17 14:27:08.811944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.811818    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"
Sep 17 14:27:08.811944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.811858    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5" event={"ID":"b1a93061-14ee-462f-8421-7abd69bbad6d","Type":"ContainerDied","Data":"0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1"}
Sep 17 14:27:08.811944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.811906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5" event={"ID":"b1a93061-14ee-462f-8421-7abd69bbad6d","Type":"ContainerDied","Data":"373e35372837647f26073cbbbdc8bbed2a587e950a87dcc0c817bc5ec81ecb76"}
Sep 17 14:27:08.811944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.811927    2565 scope.go:117] "RemoveContainer" containerID="0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1"
Sep 17 14:27:08.821847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.821827    2565 scope.go:117] "RemoveContainer" containerID="0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1"
Sep 17 14:27:08.822134 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:27:08.822111    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1\": container with ID starting with 0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1 not found: ID does not exist" containerID="0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1"
Sep 17 14:27:08.822236 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.822138    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1"} err="failed to get container status \"0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1\": rpc error: code = NotFound desc = could not find container \"0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1\": container with ID starting with 0567b3b5d736fe8690800e3d38ef3d5be40e31a81585a87417c656ccbbcc70d1 not found: ID does not exist"
Sep 17 14:27:08.840919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.840896    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"]
Sep 17 14:27:08.848145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:08.848120    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-0e6aaa/explicit-model-0e6aaa-kserve-56dd85db6f-zv8p5"]
Sep 17 14:27:09.205940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:09.205903    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:09.520653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:09.520571    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1a93061-14ee-462f-8421-7abd69bbad6d" path="/var/lib/kubelet/pods/b1a93061-14ee-462f-8421-7abd69bbad6d/volumes"
Sep 17 14:27:10.245996 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.245966    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:10.326102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.326070    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"]
Sep 17 14:27:10.326305 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.326283    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h" podUID="434e7380-24db-42cd-b24b-15dff2c00e6d" containerName="payload-processing" containerID="cri-o://e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96" gracePeriod=30
Sep 17 14:27:10.343126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.343093    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"]
Sep 17 14:27:10.343420 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.343372    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7" podUID="69a5382a-52ce-4b24-b87e-631bd10b8533" containerName="payload-pre-processing" containerID="cri-o://382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683" gracePeriod=30
Sep 17 14:27:10.434611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.434583    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:10.582551 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.582501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:10.674149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.674111    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:27:10.677910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.677881    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:27:10.718271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.718229    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"]
Sep 17 14:27:10.725810 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.725778    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-0e6aaa-zhghn"]
Sep 17 14:27:10.737522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.737481    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:10.819124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.819007    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434e7380-24db-42cd-b24b-15dff2c00e6d-plugins-config-volume\") pod \"434e7380-24db-42cd-b24b-15dff2c00e6d\" (UID: \"434e7380-24db-42cd-b24b-15dff2c00e6d\") "
Sep 17 14:27:10.819124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.819068    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqjzb\" (UniqueName: \"kubernetes.io/projected/69a5382a-52ce-4b24-b87e-631bd10b8533-kube-api-access-nqjzb\") pod \"69a5382a-52ce-4b24-b87e-631bd10b8533\" (UID: \"69a5382a-52ce-4b24-b87e-631bd10b8533\") "
Sep 17 14:27:10.819397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.819131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a5382a-52ce-4b24-b87e-631bd10b8533-plugins-config-volume\") pod \"69a5382a-52ce-4b24-b87e-631bd10b8533\" (UID: \"69a5382a-52ce-4b24-b87e-631bd10b8533\") "
Sep 17 14:27:10.819397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.819172    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxcs2\" (UniqueName: \"kubernetes.io/projected/434e7380-24db-42cd-b24b-15dff2c00e6d-kube-api-access-pxcs2\") pod \"434e7380-24db-42cd-b24b-15dff2c00e6d\" (UID: \"434e7380-24db-42cd-b24b-15dff2c00e6d\") "
Sep 17 14:27:10.820663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.820571    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a5382a-52ce-4b24-b87e-631bd10b8533-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69a5382a-52ce-4b24-b87e-631bd10b8533" (UID: "69a5382a-52ce-4b24-b87e-631bd10b8533"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:27:10.820663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.820616    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434e7380-24db-42cd-b24b-15dff2c00e6d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "434e7380-24db-42cd-b24b-15dff2c00e6d" (UID: "434e7380-24db-42cd-b24b-15dff2c00e6d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:27:10.823825 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.823793    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a5382a-52ce-4b24-b87e-631bd10b8533-kube-api-access-nqjzb" (OuterVolumeSpecName: "kube-api-access-nqjzb") pod "69a5382a-52ce-4b24-b87e-631bd10b8533" (UID: "69a5382a-52ce-4b24-b87e-631bd10b8533"). InnerVolumeSpecName "kube-api-access-nqjzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:10.825638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.825605    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434e7380-24db-42cd-b24b-15dff2c00e6d-kube-api-access-pxcs2" (OuterVolumeSpecName: "kube-api-access-pxcs2") pod "434e7380-24db-42cd-b24b-15dff2c00e6d" (UID: "434e7380-24db-42cd-b24b-15dff2c00e6d"). InnerVolumeSpecName "kube-api-access-pxcs2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:10.830349 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.830314    2565 generic.go:356] "Generic (PLEG): container finished" podID="69a5382a-52ce-4b24-b87e-631bd10b8533" containerID="382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683" exitCode=0
Sep 17 14:27:10.830467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.830355    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"
Sep 17 14:27:10.830467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.830395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7" event={"ID":"69a5382a-52ce-4b24-b87e-631bd10b8533","Type":"ContainerDied","Data":"382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683"}
Sep 17 14:27:10.830467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.830455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7" event={"ID":"69a5382a-52ce-4b24-b87e-631bd10b8533","Type":"ContainerDied","Data":"ccfce008fcc343735ba8c19a0c01fe27259cbf9ee4d80fa6ee861c22b53686fe"}
Sep 17 14:27:10.830654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.830486    2565 scope.go:117] "RemoveContainer" containerID="382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683"
Sep 17 14:27:10.831910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.831883    2565 generic.go:356] "Generic (PLEG): container finished" podID="434e7380-24db-42cd-b24b-15dff2c00e6d" containerID="e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96" exitCode=0
Sep 17 14:27:10.832022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.831921    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"
Sep 17 14:27:10.832022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.831933    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h" event={"ID":"434e7380-24db-42cd-b24b-15dff2c00e6d","Type":"ContainerDied","Data":"e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96"}
Sep 17 14:27:10.832022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.831998    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h" event={"ID":"434e7380-24db-42cd-b24b-15dff2c00e6d","Type":"ContainerDied","Data":"f0cea423c1f6a433da56e13b215c01906082788dcf2e407b4075ebbed40bcc1f"}
Sep 17 14:27:10.840247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.840228    2565 scope.go:117] "RemoveContainer" containerID="382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683"
Sep 17 14:27:10.840677 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:27:10.840658    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683\": container with ID starting with 382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683 not found: ID does not exist" containerID="382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683"
Sep 17 14:27:10.840738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.840686    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683"} err="failed to get container status \"382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683\": rpc error: code = NotFound desc = could not find container \"382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683\": container with ID starting with 382fd024c611f922eb4829efc248f8a5395dcc5a0aa9a036a7d7303bc2df3683 not found: ID does not exist"
Sep 17 14:27:10.840738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.840701    2565 scope.go:117] "RemoveContainer" containerID="e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96"
Sep 17 14:27:10.849445 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.849423    2565 scope.go:117] "RemoveContainer" containerID="e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96"
Sep 17 14:27:10.849762 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:27:10.849737    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96\": container with ID starting with e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96 not found: ID does not exist" containerID="e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96"
Sep 17 14:27:10.849836 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.849773    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96"} err="failed to get container status \"e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96\": rpc error: code = NotFound desc = could not find container \"e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96\": container with ID starting with e6e9a03d7fd3555b8e6405538d5ed53ba0f182a6b39c75834030f2c5ade59c96 not found: ID does not exist"
Sep 17 14:27:10.873873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.873845    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:10.874999 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.874978    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"]
Sep 17 14:27:10.882448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.882422    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-0e6aaa-fdc589f45-ltq9h"]
Sep 17 14:27:10.898874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.898848    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"]
Sep 17 14:27:10.909951 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.909924    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-0e6aaa-6bcb96874b-n2jl7"]
Sep 17 14:27:10.921234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.921211    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/434e7380-24db-42cd-b24b-15dff2c00e6d-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:10.921344 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.921237    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqjzb\" (UniqueName: \"kubernetes.io/projected/69a5382a-52ce-4b24-b87e-631bd10b8533-kube-api-access-nqjzb\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:10.921344 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.921249    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a5382a-52ce-4b24-b87e-631bd10b8533-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:10.921344 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:10.921271    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxcs2\" (UniqueName: \"kubernetes.io/projected/434e7380-24db-42cd-b24b-15dff2c00e6d-kube-api-access-pxcs2\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:11.083407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.083373    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:11.192231 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.192192    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:11.521379 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.521343    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="434e7380-24db-42cd-b24b-15dff2c00e6d" path="/var/lib/kubelet/pods/434e7380-24db-42cd-b24b-15dff2c00e6d/volumes"
Sep 17 14:27:11.521754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.521733    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d2feffe-60d6-4208-b8f5-21a74ee449e1" path="/var/lib/kubelet/pods/5d2feffe-60d6-4208-b8f5-21a74ee449e1/volumes"
Sep 17 14:27:11.522135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.522116    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a5382a-52ce-4b24-b87e-631bd10b8533" path="/var/lib/kubelet/pods/69a5382a-52ce-4b24-b87e-631bd10b8533/volumes"
Sep 17 14:27:11.642734 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.642662    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:11.789632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.789599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:11.907830 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.907735    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"]
Sep 17 14:27:11.908194 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908182    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1a93061-14ee-462f-8421-7abd69bbad6d" containerName="main"
Sep 17 14:27:11.908237 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908196    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1a93061-14ee-462f-8421-7abd69bbad6d" containerName="main"
Sep 17 14:27:11.908237 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908220    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a5382a-52ce-4b24-b87e-631bd10b8533" containerName="payload-pre-processing"
Sep 17 14:27:11.908237 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908228    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a5382a-52ce-4b24-b87e-631bd10b8533" containerName="payload-pre-processing"
Sep 17 14:27:11.908237 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908235    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="434e7380-24db-42cd-b24b-15dff2c00e6d" containerName="payload-processing"
Sep 17 14:27:11.908384 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908240    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="434e7380-24db-42cd-b24b-15dff2c00e6d" containerName="payload-processing"
Sep 17 14:27:11.908384 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908320    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="434e7380-24db-42cd-b24b-15dff2c00e6d" containerName="payload-processing"
Sep 17 14:27:11.908384 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908330    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a5382a-52ce-4b24-b87e-631bd10b8533" containerName="payload-pre-processing"
Sep 17 14:27:11.908384 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.908339    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1a93061-14ee-462f-8421-7abd69bbad6d" containerName="main"
Sep 17 14:27:11.912972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.912953    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:11.921489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.921468    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-130b7aa1\""
Sep 17 14:27:11.921622 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.921547    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-130b7aa1-dockercfg-cnlhr\""
Sep 17 14:27:11.939965 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:11.939938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"]
Sep 17 14:27:12.015452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.015416    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"]
Sep 17 14:27:12.018991 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.018974    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.032239 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.032215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-plugins-config-volume\") pod \"payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj\" (UID: \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\") " pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:12.032392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.032300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9w6p\" (UniqueName: \"kubernetes.io/projected/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-kube-api-access-v9w6p\") pod \"payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj\" (UID: \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\") " pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:12.039884 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.039861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"]
Sep 17 14:27:12.133885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.133847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6s6p\" (UniqueName: \"kubernetes.io/projected/6c88b707-57fe-4463-9a43-db82f031a53c-kube-api-access-h6s6p\") pod \"payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb\" (UID: \"6c88b707-57fe-4463-9a43-db82f031a53c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.134096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.133931    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-plugins-config-volume\") pod \"payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj\" (UID: \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\") " pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:12.134096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.133956    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9w6p\" (UniqueName: \"kubernetes.io/projected/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-kube-api-access-v9w6p\") pod \"payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj\" (UID: \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\") " pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:12.134096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.133972    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c88b707-57fe-4463-9a43-db82f031a53c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb\" (UID: \"6c88b707-57fe-4463-9a43-db82f031a53c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.134780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.134752    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-plugins-config-volume\") pod \"payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj\" (UID: \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\") " pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:12.149164 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.149134    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9w6p\" (UniqueName: \"kubernetes.io/projected/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-kube-api-access-v9w6p\") pod \"payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj\" (UID: \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\") " pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:12.223780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.223680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:12.235235 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.235197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c88b707-57fe-4463-9a43-db82f031a53c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb\" (UID: \"6c88b707-57fe-4463-9a43-db82f031a53c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.235397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.235379    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6s6p\" (UniqueName: \"kubernetes.io/projected/6c88b707-57fe-4463-9a43-db82f031a53c-kube-api-access-h6s6p\") pod \"payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb\" (UID: \"6c88b707-57fe-4463-9a43-db82f031a53c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.236051 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.236027    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c88b707-57fe-4463-9a43-db82f031a53c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb\" (UID: \"6c88b707-57fe-4463-9a43-db82f031a53c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.251495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.251449    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6s6p\" (UniqueName: \"kubernetes.io/projected/6c88b707-57fe-4463-9a43-db82f031a53c-kube-api-access-h6s6p\") pod \"payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb\" (UID: \"6c88b707-57fe-4463-9a43-db82f031a53c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.329134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.329105    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.436849 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.436651    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:12.526561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.526480    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"]
Sep 17 14:27:12.529746 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:27:12.529702    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ccdd89_caa1_4a0d_8bbf_c81bd08caaa0.slice/crio-95d4098fd58370b86e79096908dac7048fc5051bdb3745a5f71a092dbaeff41a WatchSource:0}: Error finding container 95d4098fd58370b86e79096908dac7048fc5051bdb3745a5f71a092dbaeff41a: Status 404 returned error can't find the container with id 95d4098fd58370b86e79096908dac7048fc5051bdb3745a5f71a092dbaeff41a
Sep 17 14:27:12.638324 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.638295    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"]
Sep 17 14:27:12.641944 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:27:12.641922    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c88b707_57fe_4463_9a43_db82f031a53c.slice/crio-5e769223c2267cdcb5142d6ec8ec8c3aecca2bc5a8255c35342ccfc717471a1c WatchSource:0}: Error finding container 5e769223c2267cdcb5142d6ec8ec8c3aecca2bc5a8255c35342ccfc717471a1c: Status 404 returned error can't find the container with id 5e769223c2267cdcb5142d6ec8ec8c3aecca2bc5a8255c35342ccfc717471a1c
Sep 17 14:27:12.849016 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.848972    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj" event={"ID":"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0","Type":"ContainerStarted","Data":"304fba490204b4f77050455820fb165ea28a89c24e481570e5b043cdcd116628"}
Sep 17 14:27:12.849518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.849043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj" event={"ID":"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0","Type":"ContainerStarted","Data":"95d4098fd58370b86e79096908dac7048fc5051bdb3745a5f71a092dbaeff41a"}
Sep 17 14:27:12.849886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.849832    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:12.851773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.851742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb" event={"ID":"6c88b707-57fe-4463-9a43-db82f031a53c","Type":"ContainerStarted","Data":"7a0930f5845979282d5fa41fb92073005dc7dd40a2eb29e30942fe8d82bf4c6e"}
Sep 17 14:27:12.851904 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.851798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb" event={"ID":"6c88b707-57fe-4463-9a43-db82f031a53c","Type":"ContainerStarted","Data":"5e769223c2267cdcb5142d6ec8ec8c3aecca2bc5a8255c35342ccfc717471a1c"}
Sep 17 14:27:12.852431 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.852401    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:12.882106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.882054    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj" podStartSLOduration=1.882035681 podStartE2EDuration="1.882035681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:27:12.877389552 +0000 UTC m=+1219.810894749" watchObservedRunningTime="2026-09-17 14:27:12.882035681 +0000 UTC m=+1219.815540875"
Sep 17 14:27:12.906397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:12.906345    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb" podStartSLOduration=1.906328561 podStartE2EDuration="1.906328561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:27:12.905017436 +0000 UTC m=+1219.838522629" watchObservedRunningTime="2026-09-17 14:27:12.906328561 +0000 UTC m=+1219.839833754"
Sep 17 14:27:13.755862 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:13.755827    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:15.015760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:15.015728    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:16.306496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:16.306462    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:17.460894 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:17.460851    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:18.621434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:18.621397    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:19.967309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:19.967272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:20.925291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:20.925244    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:21.070741 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:21.070702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:21.342350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:21.342319    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:21.359952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:21.359922    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"]
Sep 17 14:27:21.360160 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:21.360139    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy" containerID="cri-o://61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7" gracePeriod=30
Sep 17 14:27:21.469900 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:21.469854    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:21.908448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:21.908408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:22.333525 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.333493    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"]
Sep 17 14:27:22.337110 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.337091    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.340266 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.340229    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-adae21-openshift-default-dockercfg-q8rpf\""
Sep 17 14:27:22.373316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.363731    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"]
Sep 17 14:27:22.382279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.380920    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:22.467010 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.466962    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cnw8\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-kube-api-access-5cnw8\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.467399 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.467373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-istio-token\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.467581 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.467565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-certs\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.467719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.467704    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-credential-socket\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.467882 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.467867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-envoy\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.468076 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.468060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-data\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.468225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.468211    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4506154-5c08-416d-a254-f9956c528550-istio-podinfo\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.468399 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.468384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4506154-5c08-416d-a254-f9956c528550-istiod-ca-cert\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.468539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.468525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-socket\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.496202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.496113    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.569969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4506154-5c08-416d-a254-f9956c528550-istio-podinfo\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570038    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4506154-5c08-416d-a254-f9956c528550-istiod-ca-cert\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-socket\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570167    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cnw8\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-kube-api-access-5cnw8\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570210    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-istio-token\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570245    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-certs\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570289    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-credential-socket\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570341    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-envoy\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570423    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-data\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.571316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.570888    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-data\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.573058 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.573031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4506154-5c08-416d-a254-f9956c528550-istiod-ca-cert\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.574625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.573534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-socket\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.574814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.574037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-credential-socket\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.574949 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.574281    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-certs\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.577072 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.576945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4506154-5c08-416d-a254-f9956c528550-istio-podinfo\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.578466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.578444    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-envoy\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.581910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.581881    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-istio-token\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.583027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.583003    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cnw8\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-kube-api-access-5cnw8\") pod \"e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") " pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:22.648604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:22.648529    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:23.005231 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.005136    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"]
Sep 17 14:27:23.008319 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:27:23.008292    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4506154_5c08_416d_a254_f9956c528550.slice/crio-fc1965472a4b18429a7ff15e427e27e736c59450eb0fcb8b5f3fb9df9ab81058 WatchSource:0}: Error finding container fc1965472a4b18429a7ff15e427e27e736c59450eb0fcb8b5f3fb9df9ab81058: Status 404 returned error can't find the container with id fc1965472a4b18429a7ff15e427e27e736c59450eb0fcb8b5f3fb9df9ab81058
Sep 17 14:27:23.010732 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.010697    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:23.010840 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.010783    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:23.010840 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.010826    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:23.427199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.427162    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:23.835303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.835232    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:23.916466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.916430    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk" event={"ID":"b4506154-5c08-416d-a254-f9956c528550","Type":"ContainerStarted","Data":"878d5e9cfdd0fb2fef16624644f9e6a8b5e4ae1920436fadd8eec98898841adf"}
Sep 17 14:27:23.916648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.916481    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk" event={"ID":"b4506154-5c08-416d-a254-f9956c528550","Type":"ContainerStarted","Data":"fc1965472a4b18429a7ff15e427e27e736c59450eb0fcb8b5f3fb9df9ab81058"}
Sep 17 14:27:23.945626 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:23.945566    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk" podStartSLOduration=1.9455436320000001 podStartE2EDuration="1.945543632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:27:23.942798827 +0000 UTC m=+1230.876304019" watchObservedRunningTime="2026-09-17 14:27:23.945543632 +0000 UTC m=+1230.879048836"
Sep 17 14:27:24.309423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.309333    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:24.416539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.416494    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:24.526270 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.526233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:24.646382 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.646358    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:24.650958 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.650923    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:24.657604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.657570    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:24.766318 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.766281    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:24.866637 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.866604    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:24.866836 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.866724    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:24.926190 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.924316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:24.926190 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:24.925509    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:27:25.080941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:25.080897    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:25.340210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:25.340177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:26.171853 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.171816    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:26.670767 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.670447    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:27:26.726488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726415    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-socket\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.726488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726482    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istiod-ca-cert\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.726795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726533    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-podinfo\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.726795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726568    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-certs\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.726795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726593    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-credential-socket\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.726795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726638    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-data\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.726795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726672    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-token\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.726795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726716    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cjzj\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-kube-api-access-6cjzj\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.726795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.726760    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-envoy\") pod \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\" (UID: \"feec9f8e-d138-4d04-8339-efbf73e5ff2e\") "
Sep 17 14:27:26.727228 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.727201    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:26.727610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.727411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:26.727610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.727454    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-data" (OuterVolumeSpecName: "istio-data") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:26.727791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.727624    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:26.727791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.727664    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:27:26.729596 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.729569    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:27:26.729779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.729758    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-kube-api-access-6cjzj" (OuterVolumeSpecName: "kube-api-access-6cjzj") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "kube-api-access-6cjzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:26.729853 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.729837    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:27:26.730340 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.730322    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-token" (OuterVolumeSpecName: "istio-token") pod "feec9f8e-d138-4d04-8339-efbf73e5ff2e" (UID: "feec9f8e-d138-4d04-8339-efbf73e5ff2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:26.828636 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828605    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.828636 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828631    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.828636 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828640    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.828870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828649    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.828870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828657    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cjzj\" (UniqueName: \"kubernetes.io/projected/feec9f8e-d138-4d04-8339-efbf73e5ff2e-kube-api-access-6cjzj\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.828870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828665    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.828870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828673    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feec9f8e-d138-4d04-8339-efbf73e5ff2e-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.828870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828680    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.828870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.828691    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feec9f8e-d138-4d04-8339-efbf73e5ff2e-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:26.938560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.938523    2565 generic.go:356] "Generic (PLEG): container finished" podID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerID="61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7" exitCode=0
Sep 17 14:27:26.938560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.938551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" event={"ID":"feec9f8e-d138-4d04-8339-efbf73e5ff2e","Type":"ContainerDied","Data":"61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7"}
Sep 17 14:27:26.938793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.938635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl" event={"ID":"feec9f8e-d138-4d04-8339-efbf73e5ff2e","Type":"ContainerDied","Data":"9a2b0ca8215938e7bbc611ad1368b2b6d781e376ef0ac51af9a08476fbac278e"}
Sep 17 14:27:26.938793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.938658    2565 scope.go:117] "RemoveContainer" containerID="61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7"
Sep 17 14:27:26.938962 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.938945    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"
Sep 17 14:27:26.961541 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.961518    2565 scope.go:117] "RemoveContainer" containerID="7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b"
Sep 17 14:27:26.981928 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.981823    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"]
Sep 17 14:27:26.983754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.983724    2565 scope.go:117] "RemoveContainer" containerID="61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7"
Sep 17 14:27:26.984131 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:27:26.984017    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7\": container with ID starting with 61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7 not found: ID does not exist" containerID="61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7"
Sep 17 14:27:26.984131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.984048    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7"} err="failed to get container status \"61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7\": rpc error: code = NotFound desc = could not find container \"61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7\": container with ID starting with 61b16bfe02f51dbe0cb37196a9a93f6b4e801e82ec7d0a2135c36e84a5ab74f7 not found: ID does not exist"
Sep 17 14:27:26.984131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.984065    2565 scope.go:117] "RemoveContainer" containerID="7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b"
Sep 17 14:27:26.984407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.984137    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-0e6aaa-openshift-default-6c7f796c8d-x65dl"]
Sep 17 14:27:26.984407 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:27:26.984379    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b\": container with ID starting with 7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b not found: ID does not exist" containerID="7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b"
Sep 17 14:27:26.984474 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:26.984404    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b"} err="failed to get container status \"7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b\": rpc error: code = NotFound desc = could not find container \"7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b\": container with ID starting with 7ff6ff8a71b8d9a7063a327d8eedba68dab38a143cc2c23cc579575dec1c9f3b not found: ID does not exist"
Sep 17 14:27:27.522797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:27.522709    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" path="/var/lib/kubelet/pods/feec9f8e-d138-4d04-8339-efbf73e5ff2e/volumes"
Sep 17 14:27:30.969483 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:30.969451    2565 generic.go:356] "Generic (PLEG): container finished" podID="c1cdd559-2db2-46a1-8afa-81da80143208" containerID="7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694" exitCode=0
Sep 17 14:27:30.970022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:30.969523    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" event={"ID":"c1cdd559-2db2-46a1-8afa-81da80143208","Type":"ContainerDied","Data":"7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694"}
Sep 17 14:27:30.970151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:30.970133    2565 scope.go:117] "RemoveContainer" containerID="7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694"
Sep 17 14:27:30.971455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:30.971428    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:30.971525 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:30.971501    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:30.971563 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:30.971533    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:27:31.981448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:31.981411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" event={"ID":"c1cdd559-2db2-46a1-8afa-81da80143208","Type":"ContainerStarted","Data":"e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079"}
Sep 17 14:27:32.363077 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.363036    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:32.460607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.460554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:32.577758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.577716    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:32.619718 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.619620    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:32.620688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.620663    2565 patch_prober.go:28] interesting pod/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 17 14:27:32.620756 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.620721    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 17 14:27:32.776315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.776272    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:32.810052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.810020    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"]
Sep 17 14:27:32.810529 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.810515    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy"
Sep 17 14:27:32.810598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.810531    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy"
Sep 17 14:27:32.810598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.810546    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy"
Sep 17 14:27:32.810598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.810552    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy"
Sep 17 14:27:32.810696 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.810618    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy"
Sep 17 14:27:32.810696 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.810626    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="feec9f8e-d138-4d04-8339-efbf73e5ff2e" containerName="istio-proxy"
Sep 17 14:27:32.813026 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.813008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"
Sep 17 14:27:32.815730 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.815686    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:27:32.816747 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.816698    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:27:32.835007 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.834978    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"]
Sep 17 14:27:32.895357 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.895233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d034448-3928-4fea-b9e3-31afa7d705df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz\" (UID: \"8d034448-3928-4fea-b9e3-31afa7d705df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"
Sep 17 14:27:32.996916 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.996886    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d034448-3928-4fea-b9e3-31afa7d705df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz\" (UID: \"8d034448-3928-4fea-b9e3-31afa7d705df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"
Sep 17 14:27:32.997690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:32.997664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d034448-3928-4fea-b9e3-31afa7d705df-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz\" (UID: \"8d034448-3928-4fea-b9e3-31afa7d705df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"
Sep 17 14:27:33.124107 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:33.124052    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"
Sep 17 14:27:33.443446 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:33.443413    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"]
Sep 17 14:27:33.447231 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:27:33.447157    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d034448_3928_4fea_b9e3_31afa7d705df.slice/crio-5e729cdeb4bb9d2c769b22f84e26a90884b186b0ac1a6855114ff5f652f672c5 WatchSource:0}: Error finding container 5e729cdeb4bb9d2c769b22f84e26a90884b186b0ac1a6855114ff5f652f672c5: Status 404 returned error can't find the container with id 5e729cdeb4bb9d2c769b22f84e26a90884b186b0ac1a6855114ff5f652f672c5
Sep 17 14:27:33.620115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:33.620078    2565 patch_prober.go:28] interesting pod/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 17 14:27:33.620332 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:33.620132    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 17 14:27:33.697429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:33.697316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:33.994794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:33.994688    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz" event={"ID":"8d034448-3928-4fea-b9e3-31afa7d705df","Type":"ContainerStarted","Data":"f0b187975c2aac91158f425a9a401022347fd71134579906ae1313358bf30227"}
Sep 17 14:27:33.994794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:33.994749    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz" event={"ID":"8d034448-3928-4fea-b9e3-31afa7d705df","Type":"ContainerStarted","Data":"5e729cdeb4bb9d2c769b22f84e26a90884b186b0ac1a6855114ff5f652f672c5"}
Sep 17 14:27:34.020049 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:34.019988    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz" podStartSLOduration=2.019970615 podStartE2EDuration="2.019970615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:27:34.018370653 +0000 UTC m=+1240.951875844" watchObservedRunningTime="2026-09-17 14:27:34.019970615 +0000 UTC m=+1240.953475800"
Sep 17 14:27:34.619846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:34.619747    2565 patch_prober.go:28] interesting pod/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 17 14:27:34.619846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:34.619812    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 17 14:27:34.682844 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:34.682712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:35.002793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:35.002712    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d034448-3928-4fea-b9e3-31afa7d705df" containerID="f0b187975c2aac91158f425a9a401022347fd71134579906ae1313358bf30227" exitCode=0
Sep 17 14:27:35.002948 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:35.002797    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz" event={"ID":"8d034448-3928-4fea-b9e3-31afa7d705df","Type":"ContainerDied","Data":"f0b187975c2aac91158f425a9a401022347fd71134579906ae1313358bf30227"}
Sep 17 14:27:35.620038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:35.619998    2565 patch_prober.go:28] interesting pod/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 17 14:27:35.620436 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:35.620072    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 17 14:27:35.650421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:35.650381    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:36.194199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:36.194166    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"
Sep 17 14:27:36.231687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:36.231657    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d034448-3928-4fea-b9e3-31afa7d705df-maas-api-service-ca\") pod \"8d034448-3928-4fea-b9e3-31afa7d705df\" (UID: \"8d034448-3928-4fea-b9e3-31afa7d705df\") "
Sep 17 14:27:36.232094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:36.232067    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d034448-3928-4fea-b9e3-31afa7d705df-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d034448-3928-4fea-b9e3-31afa7d705df" (UID: "8d034448-3928-4fea-b9e3-31afa7d705df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:27:36.332807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:36.332779    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d034448-3928-4fea-b9e3-31afa7d705df-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:36.488813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:36.488782    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:36.620017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:36.619935    2565 patch_prober.go:28] interesting pod/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 17 14:27:36.620017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:36.619990    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 17 14:27:37.017881 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:37.017800    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"
Sep 17 14:27:37.018029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:37.017800    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz" event={"ID":"8d034448-3928-4fea-b9e3-31afa7d705df","Type":"ContainerDied","Data":"5e729cdeb4bb9d2c769b22f84e26a90884b186b0ac1a6855114ff5f652f672c5"}
Sep 17 14:27:37.018029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:37.017929    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e729cdeb4bb9d2c769b22f84e26a90884b186b0ac1a6855114ff5f652f672c5"
Sep 17 14:27:37.467151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:37.467119    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:37.624281 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:37.624224    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:37.625332 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:37.625310    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:27:38.212474 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:38.212442    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:40.075023 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.074983    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"]
Sep 17 14:27:40.075697 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.075677    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d034448-3928-4fea-b9e3-31afa7d705df" containerName="revoke-keys"
Sep 17 14:27:40.075749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.075700    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d034448-3928-4fea-b9e3-31afa7d705df" containerName="revoke-keys"
Sep 17 14:27:40.075803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.075794    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d034448-3928-4fea-b9e3-31afa7d705df" containerName="revoke-keys"
Sep 17 14:27:40.078268 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.078236    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.081846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.081823    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"e2e-conflict-a-4102a62c-kserve-self-signed-certs\""
Sep 17 14:27:40.103896 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.103871    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"]
Sep 17 14:27:40.163452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.163424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:40.176709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.176679    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tmp-dir\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.176872 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.176801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-dshm\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.176872 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.176829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tls-certs\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.176971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.176871    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-model-cache\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.176971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.176917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpj4k\" (UniqueName: \"kubernetes.io/projected/bc5096ed-0d69-4616-8f9f-3663dfbf940d-kube-api-access-cpj4k\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.176971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.176963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-home\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.277529 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.277499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-dshm\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.277720 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.277541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tls-certs\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.277720 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.277566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-model-cache\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.277720 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.277612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpj4k\" (UniqueName: \"kubernetes.io/projected/bc5096ed-0d69-4616-8f9f-3663dfbf940d-kube-api-access-cpj4k\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.277720 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.277636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-home\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.277961 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.277809    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tmp-dir\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.278047 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.278026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-home\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.278111 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.278085    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-model-cache\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.278274 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.278227    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tmp-dir\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.279981 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.279950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-dshm\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.280140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.280120    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tls-certs\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.286771 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.286749    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpj4k\" (UniqueName: \"kubernetes.io/projected/bc5096ed-0d69-4616-8f9f-3663dfbf940d-kube-api-access-cpj4k\") pod \"e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") " pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.388788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.388714    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:40.665186 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:40.665159    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"]
Sep 17 14:27:40.666095 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:27:40.666072    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc5096ed_0d69_4616_8f9f_3663dfbf940d.slice/crio-04afdc5fca247b7c27ad88418204f1b7db101c574e6996c66b6f4396300ceb29 WatchSource:0}: Error finding container 04afdc5fca247b7c27ad88418204f1b7db101c574e6996c66b6f4396300ceb29: Status 404 returned error can't find the container with id 04afdc5fca247b7c27ad88418204f1b7db101c574e6996c66b6f4396300ceb29
Sep 17 14:27:41.043116 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:41.043028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs" event={"ID":"bc5096ed-0d69-4616-8f9f-3663dfbf940d","Type":"ContainerStarted","Data":"134784e14dd5e181fcc1bb5f132596da9a3b30d83af2c0908f8fc862b66ef3dd"}
Sep 17 14:27:41.043116 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:41.043074    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs" event={"ID":"bc5096ed-0d69-4616-8f9f-3663dfbf940d","Type":"ContainerStarted","Data":"04afdc5fca247b7c27ad88418204f1b7db101c574e6996c66b6f4396300ceb29"}
Sep 17 14:27:41.075389 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:41.075337    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs" podStartSLOduration=1.075322715 podStartE2EDuration="1.075322715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:27:41.074280948 +0000 UTC m=+1248.007786137" watchObservedRunningTime="2026-09-17 14:27:41.075322715 +0000 UTC m=+1248.008827906"
Sep 17 14:27:43.571212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:43.571179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:45.964460 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:45.964430    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:46.839893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:46.839863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:46.875181 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:46.875147    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"]
Sep 17 14:27:46.875541 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:46.875508    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj" podUID="57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0" containerName="payload-processing" containerID="cri-o://304fba490204b4f77050455820fb165ea28a89c24e481570e5b043cdcd116628" gracePeriod=30
Sep 17 14:27:46.888499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:46.888387    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"]
Sep 17 14:27:46.888689 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:46.888640    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb" podUID="6c88b707-57fe-4463-9a43-db82f031a53c" containerName="payload-pre-processing" containerID="cri-o://7a0930f5845979282d5fa41fb92073005dc7dd40a2eb29e30942fe8d82bf4c6e" gracePeriod=30
Sep 17 14:27:46.939074 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:46.939031    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:47.051044 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.050996    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:47.083355 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.083324    2565 generic.go:356] "Generic (PLEG): container finished" podID="57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0" containerID="304fba490204b4f77050455820fb165ea28a89c24e481570e5b043cdcd116628" exitCode=0
Sep 17 14:27:47.083515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.083475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj" event={"ID":"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0","Type":"ContainerDied","Data":"304fba490204b4f77050455820fb165ea28a89c24e481570e5b043cdcd116628"}
Sep 17 14:27:47.085172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.085147    2565 generic.go:356] "Generic (PLEG): container finished" podID="6c88b707-57fe-4463-9a43-db82f031a53c" containerID="7a0930f5845979282d5fa41fb92073005dc7dd40a2eb29e30942fe8d82bf4c6e" exitCode=0
Sep 17 14:27:47.085322 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.085182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb" event={"ID":"6c88b707-57fe-4463-9a43-db82f031a53c","Type":"ContainerDied","Data":"7a0930f5845979282d5fa41fb92073005dc7dd40a2eb29e30942fe8d82bf4c6e"}
Sep 17 14:27:47.096375 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.096349    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"]
Sep 17 14:27:47.100095 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.100071    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-130b7aa1-sd8zz"]
Sep 17 14:27:47.151223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.151165    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:47.178604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.178584    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:47.181712 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.181692    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:47.256029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.255990    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-plugins-config-volume\") pod \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\" (UID: \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\") "
Sep 17 14:27:47.256238 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.256055    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c88b707-57fe-4463-9a43-db82f031a53c-plugins-config-volume\") pod \"6c88b707-57fe-4463-9a43-db82f031a53c\" (UID: \"6c88b707-57fe-4463-9a43-db82f031a53c\") "
Sep 17 14:27:47.256238 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.256151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6s6p\" (UniqueName: \"kubernetes.io/projected/6c88b707-57fe-4463-9a43-db82f031a53c-kube-api-access-h6s6p\") pod \"6c88b707-57fe-4463-9a43-db82f031a53c\" (UID: \"6c88b707-57fe-4463-9a43-db82f031a53c\") "
Sep 17 14:27:47.256238 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.256203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9w6p\" (UniqueName: \"kubernetes.io/projected/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-kube-api-access-v9w6p\") pod \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\" (UID: \"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0\") "
Sep 17 14:27:47.257923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.257488    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0" (UID: "57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:27:47.257923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.257699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c88b707-57fe-4463-9a43-db82f031a53c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c88b707-57fe-4463-9a43-db82f031a53c" (UID: "6c88b707-57fe-4463-9a43-db82f031a53c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:27:47.259989 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.259959    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-kube-api-access-v9w6p" (OuterVolumeSpecName: "kube-api-access-v9w6p") pod "57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0" (UID: "57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0"). InnerVolumeSpecName "kube-api-access-v9w6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:47.260201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.260173    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c88b707-57fe-4463-9a43-db82f031a53c-kube-api-access-h6s6p" (OuterVolumeSpecName: "kube-api-access-h6s6p") pod "6c88b707-57fe-4463-9a43-db82f031a53c" (UID: "6c88b707-57fe-4463-9a43-db82f031a53c"). InnerVolumeSpecName "kube-api-access-h6s6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:27:47.358328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.358224    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6s6p\" (UniqueName: \"kubernetes.io/projected/6c88b707-57fe-4463-9a43-db82f031a53c-kube-api-access-h6s6p\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:47.358328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.358267    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9w6p\" (UniqueName: \"kubernetes.io/projected/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-kube-api-access-v9w6p\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:47.358328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.358277    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:47.358328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.358289    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c88b707-57fe-4463-9a43-db82f031a53c-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:27:47.486123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.486092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:47.521141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.521107    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d034448-3928-4fea-b9e3-31afa7d705df" path="/var/lib/kubelet/pods/8d034448-3928-4fea-b9e3-31afa7d705df/volumes"
Sep 17 14:27:47.704904 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:47.704819    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:48.091800 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.091771    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"
Sep 17 14:27:48.092209 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.091790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj" event={"ID":"57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0","Type":"ContainerDied","Data":"95d4098fd58370b86e79096908dac7048fc5051bdb3745a5f71a092dbaeff41a"}
Sep 17 14:27:48.092209 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.091881    2565 scope.go:117] "RemoveContainer" containerID="304fba490204b4f77050455820fb165ea28a89c24e481570e5b043cdcd116628"
Sep 17 14:27:48.093397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.093380    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"
Sep 17 14:27:48.093397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.093382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb" event={"ID":"6c88b707-57fe-4463-9a43-db82f031a53c","Type":"ContainerDied","Data":"5e769223c2267cdcb5142d6ec8ec8c3aecca2bc5a8255c35342ccfc717471a1c"}
Sep 17 14:27:48.100777 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.100758    2565 scope.go:117] "RemoveContainer" containerID="7a0930f5845979282d5fa41fb92073005dc7dd40a2eb29e30942fe8d82bf4c6e"
Sep 17 14:27:48.113802 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.113780    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"]
Sep 17 14:27:48.117554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.117531    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-130b7aa1-6855bb4f98-96kkb"]
Sep 17 14:27:48.130121 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.130097    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"]
Sep 17 14:27:48.136623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.136603    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-130b7aa1-7db599d5cc-l6zgj"]
Sep 17 14:27:48.573208 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:48.573183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:49.342517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:49.342486    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:49.520131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:49.520090    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0" path="/var/lib/kubelet/pods/57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0/volumes"
Sep 17 14:27:49.520618 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:49.520595    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c88b707-57fe-4463-9a43-db82f031a53c" path="/var/lib/kubelet/pods/6c88b707-57fe-4463-9a43-db82f031a53c/volumes"
Sep 17 14:27:50.208909 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:50.208871    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:50.389801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:50.389762    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:50.389801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:50.389806    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:50.403010 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:50.402978    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:51.124572 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:51.124538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:27:51.206408 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:51.206371    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:54.390674 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:54.390635    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:54.502682 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:54.502646    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:54.715967 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:54.715878    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"]
Sep 17 14:27:54.716138 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:54.716096    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy" containerID="cri-o://e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079" gracePeriod=30
Sep 17 14:27:54.784594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:54.784551    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:55.212026 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.211991    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:55.377608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.377567    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:55.486591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.486502    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:55.678974 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.678935    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"]
Sep 17 14:27:55.679419 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.679404    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c88b707-57fe-4463-9a43-db82f031a53c" containerName="payload-pre-processing"
Sep 17 14:27:55.679467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.679422    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c88b707-57fe-4463-9a43-db82f031a53c" containerName="payload-pre-processing"
Sep 17 14:27:55.679467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.679446    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0" containerName="payload-processing"
Sep 17 14:27:55.679467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.679452    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0" containerName="payload-processing"
Sep 17 14:27:55.679572 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.679530    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="57ccdd89-caa1-4a0d-8bbf-c81bd08caaa0" containerName="payload-processing"
Sep 17 14:27:55.679572 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.679543    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c88b707-57fe-4463-9a43-db82f031a53c" containerName="payload-pre-processing"
Sep 17 14:27:55.684488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.684470    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.689809 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.689787    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-080801\"/\"default-dockercfg-8mcjd\""
Sep 17 14:27:55.689998 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.689980    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-080801\"/\"distinct-w0-080801-kserve-self-signed-certs\""
Sep 17 14:27:55.691380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.691012    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-080801\"/\"kube-root-ca.crt\""
Sep 17 14:27:55.691380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.691027    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-080801\"/\"openshift-service-ca.crt\""
Sep 17 14:27:55.709005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.708970    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"]
Sep 17 14:27:55.746006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.745932    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tmp-dir\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.746006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.745982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tls-certs\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.746224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.746045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-dshm\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.746224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.746110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47kj\" (UniqueName: \"kubernetes.io/projected/bb2e1881-ce05-467b-be0c-5dc0d27b899f-kube-api-access-w47kj\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.746224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.746143    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-home\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.746224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.746170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-model-cache\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tmp-dir\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tls-certs\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847113    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-dshm\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47kj\" (UniqueName: \"kubernetes.io/projected/bb2e1881-ce05-467b-be0c-5dc0d27b899f-kube-api-access-w47kj\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-home\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-model-cache\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847650 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847624    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tmp-dir\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847650 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-home\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.847750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.847682    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-model-cache\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.849796 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.849756    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-dshm\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.850165 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.850143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tls-certs\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.877471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.877442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47kj\" (UniqueName: \"kubernetes.io/projected/bb2e1881-ce05-467b-be0c-5dc0d27b899f-kube-api-access-w47kj\") pod \"distinct-w0-080801-kserve-9cbdb9576-zd6cf\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") " pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:55.965617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.965587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:55.993828 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:55.993793    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:27:56.289398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:56.289368    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"]
Sep 17 14:27:56.291539 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:27:56.291513    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb2e1881_ce05_467b_be0c_5dc0d27b899f.slice/crio-2867dc01368c750cb62645146564e1424077d2caf60f3aa3130bbb8539061465 WatchSource:0}: Error finding container 2867dc01368c750cb62645146564e1424077d2caf60f3aa3130bbb8539061465: Status 404 returned error can't find the container with id 2867dc01368c750cb62645146564e1424077d2caf60f3aa3130bbb8539061465
Sep 17 14:27:57.149097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:57.149056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf" event={"ID":"bb2e1881-ce05-467b-be0c-5dc0d27b899f","Type":"ContainerStarted","Data":"5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a"}
Sep 17 14:27:57.149589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:57.149107    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf" event={"ID":"bb2e1881-ce05-467b-be0c-5dc0d27b899f","Type":"ContainerStarted","Data":"2867dc01368c750cb62645146564e1424077d2caf60f3aa3130bbb8539061465"}
Sep 17 14:27:57.205091 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:57.205041    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf" podStartSLOduration=2.205026974 podStartE2EDuration="2.205026974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:27:57.204324799 +0000 UTC m=+1264.137829991" watchObservedRunningTime="2026-09-17 14:27:57.205026974 +0000 UTC m=+1264.138532165"
Sep 17 14:27:58.811272 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:58.811212    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:27:59.986040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:27:59.985964    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:28:00.103264 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103227    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-envoy\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103296    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-istio-token\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103319    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-certs\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103377    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpqbf\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-kube-api-access-wpqbf\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103418    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-credential-socket\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103466    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-socket\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103504    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-data\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103534    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cdd559-2db2-46a1-8afa-81da80143208-istio-podinfo\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103613    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cdd559-2db2-46a1-8afa-81da80143208-istiod-ca-cert\") pod \"c1cdd559-2db2-46a1-8afa-81da80143208\" (UID: \"c1cdd559-2db2-46a1-8afa-81da80143208\") "
Sep 17 14:28:00.103875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103644    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:00.103875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103721    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:00.103875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103745    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:00.103875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.103803    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:00.104100 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.104069    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cdd559-2db2-46a1-8afa-81da80143208-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:28:00.104167 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.104116    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:00.104167 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.104136    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:00.104167 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.104152    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:00.104167 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.104165    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:00.105742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.105716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:00.105856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.105723    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-kube-api-access-wpqbf" (OuterVolumeSpecName: "kube-api-access-wpqbf") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "kube-api-access-wpqbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:28:00.106018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.105992    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1cdd559-2db2-46a1-8afa-81da80143208-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:28:00.106118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.106001    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1cdd559-2db2-46a1-8afa-81da80143208" (UID: "c1cdd559-2db2-46a1-8afa-81da80143208"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:28:00.170321 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.170287    2565 generic.go:356] "Generic (PLEG): container finished" podID="c1cdd559-2db2-46a1-8afa-81da80143208" containerID="e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079" exitCode=0
Sep 17 14:28:00.170479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.170339    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"
Sep 17 14:28:00.170479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.170377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" event={"ID":"c1cdd559-2db2-46a1-8afa-81da80143208","Type":"ContainerDied","Data":"e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079"}
Sep 17 14:28:00.170479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.170437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7" event={"ID":"c1cdd559-2db2-46a1-8afa-81da80143208","Type":"ContainerDied","Data":"d83e349a4726b7b8905bb07396d4236712114c24dc63980b31580f8d13695024"}
Sep 17 14:28:00.170479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.170467    2565 scope.go:117] "RemoveContainer" containerID="e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079"
Sep 17 14:28:00.180126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.180111    2565 scope.go:117] "RemoveContainer" containerID="7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694"
Sep 17 14:28:00.188645 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.188626    2565 scope.go:117] "RemoveContainer" containerID="e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079"
Sep 17 14:28:00.188912 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:28:00.188893    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079\": container with ID starting with e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079 not found: ID does not exist" containerID="e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079"
Sep 17 14:28:00.188983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.188921    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079"} err="failed to get container status \"e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079\": rpc error: code = NotFound desc = could not find container \"e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079\": container with ID starting with e48071aceadbdf9d23f7bb25ecbde39a5bd91f29b7097909708b85843f992079 not found: ID does not exist"
Sep 17 14:28:00.188983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.188939    2565 scope.go:117] "RemoveContainer" containerID="7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694"
Sep 17 14:28:00.189217 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:28:00.189199    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694\": container with ID starting with 7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694 not found: ID does not exist" containerID="7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694"
Sep 17 14:28:00.189282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.189224    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694"} err="failed to get container status \"7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694\": rpc error: code = NotFound desc = could not find container \"7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694\": container with ID starting with 7172c9c510020cd926214de328dd40dd885603f4cb085c2d8682726cd9d7b694 not found: ID does not exist"
Sep 17 14:28:00.199117 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.199054    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"]
Sep 17 14:28:00.201441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.201418    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-130b7aa1-openshift-default-7855464968-kqdt7"]
Sep 17 14:28:00.205361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.205342    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpqbf\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-kube-api-access-wpqbf\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:00.205410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.205364    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cdd559-2db2-46a1-8afa-81da80143208-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:00.205410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.205373    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cdd559-2db2-46a1-8afa-81da80143208-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:00.205410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.205382    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cdd559-2db2-46a1-8afa-81da80143208-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:00.205410 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:00.205390    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cdd559-2db2-46a1-8afa-81da80143208-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:01.520274 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:01.520205    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" path="/var/lib/kubelet/pods/c1cdd559-2db2-46a1-8afa-81da80143208/volumes"
Sep 17 14:28:02.297854 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:02.297815    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:02.556402 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:02.556317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:02.743959 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:02.743923    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:02.861906 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:02.861872    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:05.994971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:05.994929    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:28:05.995441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:05.994987    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:28:06.007715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:06.007685    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:28:06.217415 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:06.217377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:28:09.374977 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:09.374925    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:09.484453 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:09.484412    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:10.286138 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.286102    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:10.437333 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.437298    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:10.584725 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.584683    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"]
Sep 17 14:28:10.585370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.585348    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy"
Sep 17 14:28:10.585436 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.585374    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy"
Sep 17 14:28:10.585436 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.585404    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy"
Sep 17 14:28:10.585436 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.585413    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy"
Sep 17 14:28:10.585547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.585535    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy"
Sep 17 14:28:10.585583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.585553    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1cdd559-2db2-46a1-8afa-81da80143208" containerName="istio-proxy"
Sep 17 14:28:10.590581 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.590562    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.592905 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.592889    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4009da-openshift-default-dockercfg-btj89\""
Sep 17 14:28:10.601559 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.601533    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"]
Sep 17 14:28:10.628583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.628548    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:10.721910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.721862    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-credential-socket\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.722127 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.721971    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-podinfo\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.722127 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.722002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-envoy\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.722127 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.722068    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-data\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.722323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.722160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-socket\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.722323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.722282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.722429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.722329    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-certs\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.722429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.722415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-token\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.722530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.722469    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqq8w\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-kube-api-access-fqq8w\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.735314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.735288    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:10.823308 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823271    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqq8w\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-kube-api-access-fqq8w\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-credential-socket\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-podinfo\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823414    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-envoy\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823439    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-data\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823476    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-socket\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823496    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-certs\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823728    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-credential-socket\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823765    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-token\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.823864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823807    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-data\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.824030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823876    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-socket\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.824030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.823965    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-certs\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.824158 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.824139    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istiod-ca-cert\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.825742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.825713    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-envoy\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.825948 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.825931    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-podinfo\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.831633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.831609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-token\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.832032 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.832011    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqq8w\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-kube-api-access-fqq8w\") pod \"e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") " pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:10.903008 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:10.902924    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:11.184737 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:11.184706    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"]
Sep 17 14:28:11.185278 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:28:11.185225    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d6e611_9d6f_49dc_82b5_2a3ded3781d3.slice/crio-ce1e38b00f1e4d392115c289922d1cd76618090b1a73c05d9471aae6852295f3 WatchSource:0}: Error finding container ce1e38b00f1e4d392115c289922d1cd76618090b1a73c05d9471aae6852295f3: Status 404 returned error can't find the container with id ce1e38b00f1e4d392115c289922d1cd76618090b1a73c05d9471aae6852295f3
Sep 17 14:28:11.187822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:11.187790    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:11.187897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:11.187857    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:11.187897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:11.187885    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:11.235541 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:11.235517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" event={"ID":"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3","Type":"ContainerStarted","Data":"ce1e38b00f1e4d392115c289922d1cd76618090b1a73c05d9471aae6852295f3"}
Sep 17 14:28:12.243216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:12.243180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" event={"ID":"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3","Type":"ContainerStarted","Data":"25ec8c694bc1aebf5b7275314880065b56b433a76d758bf7136042e28f5db628"}
Sep 17 14:28:12.270052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:12.269998    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podStartSLOduration=2.269982277 podStartE2EDuration="2.269982277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:28:12.268742375 +0000 UTC m=+1279.202247568" watchObservedRunningTime="2026-09-17 14:28:12.269982277 +0000 UTC m=+1279.203487469"
Sep 17 14:28:12.903222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:12.903186    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:12.908711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:12.908683    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:13.250163 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:13.249992    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:13.251063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:13.251041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:14.170175 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:14.170128    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:14.324602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:14.324565    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:14.427835 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:14.427739    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:15.773346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:15.773311    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:16.701010 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:16.700978    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:17.685731 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:17.685694    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:18.684976 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:18.684938    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:18.808479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:18.808442    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:18.911108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:18.911074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:19.414423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:19.414392    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:19.571175 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:19.571136    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:27.144479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:27.144439    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"]
Sep 17 14:28:27.195603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:27.195570    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:42.102313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.102276    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:42.167963 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.167913    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz" podUID="66468531-68b5-4298-b30a-80770b511f95" containerName="main" containerID="cri-o://2ae0622b98081d1c19c21b6cd62aff452c039a3de7a07499940c0568592204ae" gracePeriod=45
Sep 17 14:28:42.304220 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.304187    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"]
Sep 17 14:28:42.304481 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.304434    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk" podUID="b4506154-5c08-416d-a254-f9956c528550" containerName="istio-proxy" containerID="cri-o://878d5e9cfdd0fb2fef16624644f9e6a8b5e4ae1920436fadd8eec98898841adf" gracePeriod=30
Sep 17 14:28:42.445710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.445674    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:42.447213 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.447177    2565 generic.go:356] "Generic (PLEG): container finished" podID="66468531-68b5-4298-b30a-80770b511f95" containerID="2ae0622b98081d1c19c21b6cd62aff452c039a3de7a07499940c0568592204ae" exitCode=0
Sep 17 14:28:42.447386 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.447225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz" event={"ID":"66468531-68b5-4298-b30a-80770b511f95","Type":"ContainerDied","Data":"2ae0622b98081d1c19c21b6cd62aff452c039a3de7a07499940c0568592204ae"}
Sep 17 14:28:42.504234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.504209    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:28:42.611740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.611703    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-model-cache\") pod \"66468531-68b5-4298-b30a-80770b511f95\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") "
Sep 17 14:28:42.611948 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.611749    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-dshm\") pod \"66468531-68b5-4298-b30a-80770b511f95\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") "
Sep 17 14:28:42.611948 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.611811    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-home\") pod \"66468531-68b5-4298-b30a-80770b511f95\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") "
Sep 17 14:28:42.611948 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.611861    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468531-68b5-4298-b30a-80770b511f95-tls-certs\") pod \"66468531-68b5-4298-b30a-80770b511f95\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") "
Sep 17 14:28:42.612128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.611968    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dgr4\" (UniqueName: \"kubernetes.io/projected/66468531-68b5-4298-b30a-80770b511f95-kube-api-access-7dgr4\") pod \"66468531-68b5-4298-b30a-80770b511f95\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") "
Sep 17 14:28:42.612128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.611989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-model-cache" (OuterVolumeSpecName: "model-cache") pod "66468531-68b5-4298-b30a-80770b511f95" (UID: "66468531-68b5-4298-b30a-80770b511f95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:42.612128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.612005    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-tmp-dir\") pod \"66468531-68b5-4298-b30a-80770b511f95\" (UID: \"66468531-68b5-4298-b30a-80770b511f95\") "
Sep 17 14:28:42.612128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.612100    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-home" (OuterVolumeSpecName: "home") pod "66468531-68b5-4298-b30a-80770b511f95" (UID: "66468531-68b5-4298-b30a-80770b511f95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:42.612376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.612218    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66468531-68b5-4298-b30a-80770b511f95" (UID: "66468531-68b5-4298-b30a-80770b511f95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:42.612522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.612475    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:42.612522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.612495    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:42.612522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.612503    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:42.614073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.614051    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66468531-68b5-4298-b30a-80770b511f95-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66468531-68b5-4298-b30a-80770b511f95" (UID: "66468531-68b5-4298-b30a-80770b511f95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:28:42.614153 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.614107    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66468531-68b5-4298-b30a-80770b511f95-kube-api-access-7dgr4" (OuterVolumeSpecName: "kube-api-access-7dgr4") pod "66468531-68b5-4298-b30a-80770b511f95" (UID: "66468531-68b5-4298-b30a-80770b511f95"). InnerVolumeSpecName "kube-api-access-7dgr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:28:42.614549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.614533    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-dshm" (OuterVolumeSpecName: "dshm") pod "66468531-68b5-4298-b30a-80770b511f95" (UID: "66468531-68b5-4298-b30a-80770b511f95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:42.700980 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.700930    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:42.713360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.713325    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dgr4\" (UniqueName: \"kubernetes.io/projected/66468531-68b5-4298-b30a-80770b511f95-kube-api-access-7dgr4\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:42.713360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.713355    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66468531-68b5-4298-b30a-80770b511f95-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:42.713569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.713377    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66468531-68b5-4298-b30a-80770b511f95-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:42.946485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:42.946383    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:43.467150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:43.467116    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"
Sep 17 14:28:43.467150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:43.467142    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz" event={"ID":"66468531-68b5-4298-b30a-80770b511f95","Type":"ContainerDied","Data":"dcd81d59b5b8f016b9134e6e10267601cbc25034510f3149c959d46b0c7ef833"}
Sep 17 14:28:43.467738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:43.467197    2565 scope.go:117] "RemoveContainer" containerID="2ae0622b98081d1c19c21b6cd62aff452c039a3de7a07499940c0568592204ae"
Sep 17 14:28:43.545571 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:43.545480    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"]
Sep 17 14:28:43.570970 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:43.570939    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2e1b82/test-model-2e1b82-kserve-77465b55bf-qp4tz"]
Sep 17 14:28:43.897501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:43.897450    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:44.487464 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:44.487431    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:44.900222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:44.900188    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:45.504288 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.504241    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"]
Sep 17 14:28:45.504828 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.504679    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66468531-68b5-4298-b30a-80770b511f95" containerName="main"
Sep 17 14:28:45.504828 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.504698    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="66468531-68b5-4298-b30a-80770b511f95" containerName="main"
Sep 17 14:28:45.504828 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.504781    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="66468531-68b5-4298-b30a-80770b511f95" containerName="main"
Sep 17 14:28:45.509192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.509168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.512222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.512203    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-60bf73-openshift-default-dockercfg-bjtss\""
Sep 17 14:28:45.520963 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.520936    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66468531-68b5-4298-b30a-80770b511f95" path="/var/lib/kubelet/pods/66468531-68b5-4298-b30a-80770b511f95/volumes"
Sep 17 14:28:45.543140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-istio-token\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.543336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-socket\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.543336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-certs\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.543336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543200    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-data\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.543336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543227    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-envoy\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.543336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543299    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71952167-bbbe-4528-a9e1-5c79de58326b-istiod-ca-cert\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.543336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71952167-bbbe-4528-a9e1-5c79de58326b-istio-podinfo\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.543594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543387    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-credential-socket\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.543594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.543487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swm2j\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-kube-api-access-swm2j\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.644747 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.644705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swm2j\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-kube-api-access-swm2j\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.644955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.644780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-istio-token\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.644955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.644829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-socket\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.644955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.644859    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-certs\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.644976    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-data\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-envoy\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71952167-bbbe-4528-a9e1-5c79de58326b-istiod-ca-cert\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645162    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71952167-bbbe-4528-a9e1-5c79de58326b-istio-podinfo\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-credential-socket\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645316    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-socket\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645367    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-certs\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-data\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645605 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-credential-socket\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.645923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.645897    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71952167-bbbe-4528-a9e1-5c79de58326b-istiod-ca-cert\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.647368 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.647337    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-envoy\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.648229 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.648206    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71952167-bbbe-4528-a9e1-5c79de58326b-istio-podinfo\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.673176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.673138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-istio-token\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.685379 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.685351    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swm2j\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-kube-api-access-swm2j\") pod \"e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") " pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.685905 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.685881    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"]
Sep 17 14:28:45.820811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.820715    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:45.944997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:45.944949    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:46.158405 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.158379    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"]
Sep 17 14:28:46.161416 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:28:46.161371    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71952167_bbbe_4528_a9e1_5c79de58326b.slice/crio-2c5af91cd9e4959c36ef6e660ea6cae18389f10d7d5247630ade9832265bb3f0 WatchSource:0}: Error finding container 2c5af91cd9e4959c36ef6e660ea6cae18389f10d7d5247630ade9832265bb3f0: Status 404 returned error can't find the container with id 2c5af91cd9e4959c36ef6e660ea6cae18389f10d7d5247630ade9832265bb3f0
Sep 17 14:28:46.163903 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.163868    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:46.164048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.163937    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:46.164048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.163963    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:46.268381 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.268342    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:46.497695 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.497645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" event={"ID":"71952167-bbbe-4528-a9e1-5c79de58326b","Type":"ContainerStarted","Data":"8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1"}
Sep 17 14:28:46.499309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.497726    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" event={"ID":"71952167-bbbe-4528-a9e1-5c79de58326b","Type":"ContainerStarted","Data":"2c5af91cd9e4959c36ef6e660ea6cae18389f10d7d5247630ade9832265bb3f0"}
Sep 17 14:28:46.575372 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.575319    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" podStartSLOduration=1.5752725459999999 podStartE2EDuration="1.575272546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:28:46.566814343 +0000 UTC m=+1313.500319536" watchObservedRunningTime="2026-09-17 14:28:46.575272546 +0000 UTC m=+1313.508777733"
Sep 17 14:28:46.631611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.631578    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:46.821031 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:46.820932    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:47.505773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.505737    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4506154-5c08-416d-a254-f9956c528550" containerID="878d5e9cfdd0fb2fef16624644f9e6a8b5e4ae1920436fadd8eec98898841adf" exitCode=0
Sep 17 14:28:47.505919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.505797    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk" event={"ID":"b4506154-5c08-416d-a254-f9956c528550","Type":"ContainerDied","Data":"878d5e9cfdd0fb2fef16624644f9e6a8b5e4ae1920436fadd8eec98898841adf"}
Sep 17 14:28:47.507811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.507785    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerID="25ec8c694bc1aebf5b7275314880065b56b433a76d758bf7136042e28f5db628" exitCode=0
Sep 17 14:28:47.507937 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.507852    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" event={"ID":"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3","Type":"ContainerDied","Data":"25ec8c694bc1aebf5b7275314880065b56b433a76d758bf7136042e28f5db628"}
Sep 17 14:28:47.508466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.508449    2565 scope.go:117] "RemoveContainer" containerID="25ec8c694bc1aebf5b7275314880065b56b433a76d758bf7136042e28f5db628"
Sep 17 14:28:47.509806 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.509775    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:47.509879 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.509859    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:47.509925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.509906    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:28:47.574295 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.574264    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:28:47.670558 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.670517    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-credential-socket\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.670609    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cnw8\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-kube-api-access-5cnw8\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.670639    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4506154-5c08-416d-a254-f9956c528550-istio-podinfo\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.670812    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:47.671040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.670876    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-istio-token\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.670932    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-socket\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.670996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-data\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671040 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671036    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4506154-5c08-416d-a254-f9956c528550-istiod-ca-cert\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671138    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-certs\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671160    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:47.671495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671183    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-envoy\") pod \"b4506154-5c08-416d-a254-f9956c528550\" (UID: \"b4506154-5c08-416d-a254-f9956c528550\") "
Sep 17 14:28:47.671495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4506154-5c08-416d-a254-f9956c528550-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:28:47.671729 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671702    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.671791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671729    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4506154-5c08-416d-a254-f9956c528550-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.671791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671746    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.671909 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671794    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-data" (OuterVolumeSpecName: "istio-data") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:47.671909 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.671886    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:47.673384 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.673357    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-istio-token" (OuterVolumeSpecName: "istio-token") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:28:47.673521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.673452    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-kube-api-access-5cnw8" (OuterVolumeSpecName: "kube-api-access-5cnw8") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "kube-api-access-5cnw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:28:47.673748 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.673726    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4506154-5c08-416d-a254-f9956c528550-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:28:47.674655 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.674623    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b4506154-5c08-416d-a254-f9956c528550" (UID: "b4506154-5c08-416d-a254-f9956c528550"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:28:47.690015 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.689978    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"]
Sep 17 14:28:47.690957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.690934    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4506154-5c08-416d-a254-f9956c528550" containerName="istio-proxy"
Sep 17 14:28:47.691118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.691104    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4506154-5c08-416d-a254-f9956c528550" containerName="istio-proxy"
Sep 17 14:28:47.691365 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.691349    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4506154-5c08-416d-a254-f9956c528550" containerName="istio-proxy"
Sep 17 14:28:47.695717 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.695694    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"
Sep 17 14:28:47.698630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.698606    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:28:47.699029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.699008    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:28:47.757598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.757558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"]
Sep 17 14:28:47.773289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.773226    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/438b2ad1-7864-4b7f-b6a8-8012b6524d01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z\" (UID: \"438b2ad1-7864-4b7f-b6a8-8012b6524d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"
Sep 17 14:28:47.773500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.773334    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.773500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.773350    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.773500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.773358    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cnw8\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-kube-api-access-5cnw8\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.773500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.773367    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4506154-5c08-416d-a254-f9956c528550-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.773500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.773376    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4506154-5c08-416d-a254-f9956c528550-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.773500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.773383    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4506154-5c08-416d-a254-f9956c528550-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:47.826586 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.826542    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:47.874818 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.874773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/438b2ad1-7864-4b7f-b6a8-8012b6524d01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z\" (UID: \"438b2ad1-7864-4b7f-b6a8-8012b6524d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"
Sep 17 14:28:47.875524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:47.875505    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/438b2ad1-7864-4b7f-b6a8-8012b6524d01-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z\" (UID: \"438b2ad1-7864-4b7f-b6a8-8012b6524d01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"
Sep 17 14:28:48.009857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.009825    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"
Sep 17 14:28:48.052428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.052389    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:48.330684 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.330601    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"]
Sep 17 14:28:48.339362 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:28:48.339331    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438b2ad1_7864_4b7f_b6a8_8012b6524d01.slice/crio-2d5f93843adc2ea19ab426200a111188a53b055a657534b3cb40c5a23e013266 WatchSource:0}: Error finding container 2d5f93843adc2ea19ab426200a111188a53b055a657534b3cb40c5a23e013266: Status 404 returned error can't find the container with id 2d5f93843adc2ea19ab426200a111188a53b055a657534b3cb40c5a23e013266
Sep 17 14:28:48.515805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.515755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z" event={"ID":"438b2ad1-7864-4b7f-b6a8-8012b6524d01","Type":"ContainerStarted","Data":"2d5f93843adc2ea19ab426200a111188a53b055a657534b3cb40c5a23e013266"}
Sep 17 14:28:48.517893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.517861    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" event={"ID":"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3","Type":"ContainerStarted","Data":"af6c759015edeb655a9de4a9e7f5c7b0a41b6aa96ced5a54d61331aca08c9e86"}
Sep 17 14:28:48.519368 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.519349    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"
Sep 17 14:28:48.519477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.519382    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk" event={"ID":"b4506154-5c08-416d-a254-f9956c528550","Type":"ContainerDied","Data":"fc1965472a4b18429a7ff15e427e27e736c59450eb0fcb8b5f3fb9df9ab81058"}
Sep 17 14:28:48.519477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.519419    2565 scope.go:117] "RemoveContainer" containerID="878d5e9cfdd0fb2fef16624644f9e6a8b5e4ae1920436fadd8eec98898841adf"
Sep 17 14:28:48.519658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.519639    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:48.520868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.520567    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:28:48.529631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.529605    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:48.616147 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.616056    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"]
Sep 17 14:28:48.628297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.628264    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-adae21-openshift-default-68bdbc5fb-fd7xk"]
Sep 17 14:28:48.903560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.903457    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:48.903953 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.903755    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:28:48.903953 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.903802    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 17 14:28:48.988607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:48.988572    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:49.336745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:49.336712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:49.520812 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:49.520717    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4506154-5c08-416d-a254-f9956c528550" path="/var/lib/kubelet/pods/b4506154-5c08-416d-a254-f9956c528550/volumes"
Sep 17 14:28:49.526553 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:49.526521    2565 generic.go:356] "Generic (PLEG): container finished" podID="438b2ad1-7864-4b7f-b6a8-8012b6524d01" containerID="9fe26853a0695d86ee06ee292808110338aaa8e34fbb15d10e3059d5600cf777" exitCode=0
Sep 17 14:28:49.526698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:49.526612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z" event={"ID":"438b2ad1-7864-4b7f-b6a8-8012b6524d01","Type":"ContainerDied","Data":"9fe26853a0695d86ee06ee292808110338aaa8e34fbb15d10e3059d5600cf777"}
Sep 17 14:28:49.903227 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:49.903193    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:28:49.903420 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:49.903277    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 17 14:28:50.711612 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:50.711584    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"
Sep 17 14:28:50.808558 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:50.808461    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/438b2ad1-7864-4b7f-b6a8-8012b6524d01-maas-api-service-ca\") pod \"438b2ad1-7864-4b7f-b6a8-8012b6524d01\" (UID: \"438b2ad1-7864-4b7f-b6a8-8012b6524d01\") "
Sep 17 14:28:50.808915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:50.808884    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438b2ad1-7864-4b7f-b6a8-8012b6524d01-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "438b2ad1-7864-4b7f-b6a8-8012b6524d01" (UID: "438b2ad1-7864-4b7f-b6a8-8012b6524d01"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:28:50.809005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:50.808982    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/438b2ad1-7864-4b7f-b6a8-8012b6524d01-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:28:50.904079 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:50.904042    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:28:50.904307 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:50.904106    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 17 14:28:51.414726 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:51.414687    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:51.541157 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:51.541125    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"
Sep 17 14:28:51.541335 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:51.541158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z" event={"ID":"438b2ad1-7864-4b7f-b6a8-8012b6524d01","Type":"ContainerDied","Data":"2d5f93843adc2ea19ab426200a111188a53b055a657534b3cb40c5a23e013266"}
Sep 17 14:28:51.541335 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:51.541216    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5f93843adc2ea19ab426200a111188a53b055a657534b3cb40c5a23e013266"
Sep 17 14:28:51.903751 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:51.903717    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:28:51.904153 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:51.903779    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 17 14:28:52.766336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:52.766296    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:52.904006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:52.903966    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:28:52.904534 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:52.904031    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 17 14:28:53.683233 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:53.683198    2565 scope.go:117] "RemoveContainer" containerID="966eec57251d5cd933e2ec0c03bce2f8005c3d19757589c6dbd414b027739ed1"
Sep 17 14:28:53.904423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:53.904374    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:28:53.904926 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:53.904473    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 17 14:28:54.459123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:54.459090    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:54.796954 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:54.796864    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:54.904317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:54.904243    2565 patch_prober.go:28] interesting pod/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Sep 17 14:28:54.904524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:54.904355    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Sep 17 14:28:55.924859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:55.924767    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:55.925607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:55.925571    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:55.927280 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:55.927233    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:28:55.931284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:55.931223    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:56.134618 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.134584    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:56.313431 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.313349    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"]
Sep 17 14:28:56.313877 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.313861    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438b2ad1-7864-4b7f-b6a8-8012b6524d01" containerName="revoke-keys"
Sep 17 14:28:56.313927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.313878    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="438b2ad1-7864-4b7f-b6a8-8012b6524d01" containerName="revoke-keys"
Sep 17 14:28:56.313960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.313953    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="438b2ad1-7864-4b7f-b6a8-8012b6524d01" containerName="revoke-keys"
Sep 17 14:28:56.318291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.318267    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:56.332733 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.332589    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-60bf73-dockercfg-r6jpn\""
Sep 17 14:28:56.332944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.332924    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-60bf73\""
Sep 17 14:28:56.345833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.345801    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"]
Sep 17 14:28:56.376348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.376314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz6l6\" (UniqueName: \"kubernetes.io/projected/97a45b2f-ed71-4182-b317-9527280f9ab5-kube-api-access-kz6l6\") pod \"payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq\" (UID: \"97a45b2f-ed71-4182-b317-9527280f9ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:56.376511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.376353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a45b2f-ed71-4182-b317-9527280f9ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq\" (UID: \"97a45b2f-ed71-4182-b317-9527280f9ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:56.477313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.477238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a45b2f-ed71-4182-b317-9527280f9ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq\" (UID: \"97a45b2f-ed71-4182-b317-9527280f9ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:56.477589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.477538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz6l6\" (UniqueName: \"kubernetes.io/projected/97a45b2f-ed71-4182-b317-9527280f9ab5-kube-api-access-kz6l6\") pod \"payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq\" (UID: \"97a45b2f-ed71-4182-b317-9527280f9ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:56.478049 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.478031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a45b2f-ed71-4182-b317-9527280f9ab5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq\" (UID: \"97a45b2f-ed71-4182-b317-9527280f9ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:56.500841 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.500812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz6l6\" (UniqueName: \"kubernetes.io/projected/97a45b2f-ed71-4182-b317-9527280f9ab5-kube-api-access-kz6l6\") pod \"payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq\" (UID: \"97a45b2f-ed71-4182-b317-9527280f9ab5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:56.500979 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.500872    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"]
Sep 17 14:28:56.504411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.504396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:56.528098 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.528071    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"]
Sep 17 14:28:56.579245 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.579203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366a6979-6413-4ade-831b-584ffcd9be6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv\" (UID: \"366a6979-6413-4ade-831b-584ffcd9be6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:56.579447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.579322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw2ph\" (UniqueName: \"kubernetes.io/projected/366a6979-6413-4ade-831b-584ffcd9be6e-kube-api-access-kw2ph\") pod \"payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv\" (UID: \"366a6979-6413-4ade-831b-584ffcd9be6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:56.630645 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.630615    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:56.680757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.680716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw2ph\" (UniqueName: \"kubernetes.io/projected/366a6979-6413-4ade-831b-584ffcd9be6e-kube-api-access-kw2ph\") pod \"payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv\" (UID: \"366a6979-6413-4ade-831b-584ffcd9be6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:56.681532 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.681504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366a6979-6413-4ade-831b-584ffcd9be6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv\" (UID: \"366a6979-6413-4ade-831b-584ffcd9be6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:56.683146 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.683123    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366a6979-6413-4ade-831b-584ffcd9be6e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv\" (UID: \"366a6979-6413-4ade-831b-584ffcd9be6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:56.702760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.702732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw2ph\" (UniqueName: \"kubernetes.io/projected/366a6979-6413-4ade-831b-584ffcd9be6e-kube-api-access-kw2ph\") pod \"payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv\" (UID: \"366a6979-6413-4ade-831b-584ffcd9be6e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:56.819412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.819375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:56.829528 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.829393    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:56.922103 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:56.922067    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"]
Sep 17 14:28:56.924474 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:28:56.924427    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97a45b2f_ed71_4182_b317_9527280f9ab5.slice/crio-b28c47b5dbde324bcf1af9ae894d004d75d82113aff1ed3055d2821f7fa56891 WatchSource:0}: Error finding container b28c47b5dbde324bcf1af9ae894d004d75d82113aff1ed3055d2821f7fa56891: Status 404 returned error can't find the container with id b28c47b5dbde324bcf1af9ae894d004d75d82113aff1ed3055d2821f7fa56891
Sep 17 14:28:57.126033 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:57.126006    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"]
Sep 17 14:28:57.127903 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:28:57.127868    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod366a6979_6413_4ade_831b_584ffcd9be6e.slice/crio-3f24c2fd0ba8449c6dab62d743320350d8d002b091d5024ddf6a15f587c5fc10 WatchSource:0}: Error finding container 3f24c2fd0ba8449c6dab62d743320350d8d002b091d5024ddf6a15f587c5fc10: Status 404 returned error can't find the container with id 3f24c2fd0ba8449c6dab62d743320350d8d002b091d5024ddf6a15f587c5fc10
Sep 17 14:28:57.587627 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:57.587592    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv" event={"ID":"366a6979-6413-4ade-831b-584ffcd9be6e","Type":"ContainerStarted","Data":"ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04"}
Sep 17 14:28:57.587826 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:57.587659    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv" event={"ID":"366a6979-6413-4ade-831b-584ffcd9be6e","Type":"ContainerStarted","Data":"3f24c2fd0ba8449c6dab62d743320350d8d002b091d5024ddf6a15f587c5fc10"}
Sep 17 14:28:57.587924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:57.587828    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:28:57.589496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:57.589467    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq" event={"ID":"97a45b2f-ed71-4182-b317-9527280f9ab5","Type":"ContainerStarted","Data":"91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f"}
Sep 17 14:28:57.589633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:57.589515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq" event={"ID":"97a45b2f-ed71-4182-b317-9527280f9ab5","Type":"ContainerStarted","Data":"b28c47b5dbde324bcf1af9ae894d004d75d82113aff1ed3055d2821f7fa56891"}
Sep 17 14:28:57.589746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:57.589715    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:28:57.657785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:57.657726    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv" podStartSLOduration=1.657710799 podStartE2EDuration="1.657710799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:28:57.617874023 +0000 UTC m=+1324.551379217" watchObservedRunningTime="2026-09-17 14:28:57.657710799 +0000 UTC m=+1324.591215991"
Sep 17 14:28:58.059660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:58.059569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:59.132441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:59.132405    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:59.410930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:59.410830    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:59.641773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:59.641733    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:28:59.950803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:59.950747    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq" podStartSLOduration=3.9507274629999998 podStartE2EDuration="3.950727463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:28:57.659078779 +0000 UTC m=+1324.592583970" watchObservedRunningTime="2026-09-17 14:28:59.950727463 +0000 UTC m=+1326.884232699"
Sep 17 14:28:59.952998 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:59.952971    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"]
Sep 17 14:28:59.959629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:59.959602    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-2e1b82-nzj5z"]
Sep 17 14:28:59.974561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:28:59.974527    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:00.264908 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:00.264802    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:00.441501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:00.441469    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:01.520388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:01.520355    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438b2ad1-7864-4b7f-b6a8-8012b6524d01" path="/var/lib/kubelet/pods/438b2ad1-7864-4b7f-b6a8-8012b6524d01/volumes"
Sep 17 14:29:01.553235 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:01.553201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:03.791230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:03.791188    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:05.071262 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:05.071228    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:06.921068 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:06.921025    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:07.118623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:07.118580    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:08.357244 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:08.357183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:08.598211 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:08.598181    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:29:08.598427 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:08.598275    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:29:08.733465 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:08.733373    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:08.994200 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:08.994105    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:10.136953 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:10.136919    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:11.546903 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:11.546867    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:12.214588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:12.214552    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:12.386966 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:12.386938    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:12.549819 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:12.549724    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"]
Sep 17 14:29:12.550313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:12.549970    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy" containerID="cri-o://0ce4d3c8f8846469189d32a8fade8e0b0073a02257be9e70622ee62fbd974bde" gracePeriod=30
Sep 17 14:29:12.788331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:12.788295    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:13.039777 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.039744    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"]
Sep 17 14:29:13.044850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.044827    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"
Sep 17 14:29:13.048459 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.048432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:29:13.048614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.048432    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:29:13.066747 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.066712    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2efcf2e-8f5c-4457-b048-f9022f153b2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9\" (UID: \"f2efcf2e-8f5c-4457-b048-f9022f153b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"
Sep 17 14:29:13.085967 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.085931    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"]
Sep 17 14:29:13.168062 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.168018    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2efcf2e-8f5c-4457-b048-f9022f153b2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9\" (UID: \"f2efcf2e-8f5c-4457-b048-f9022f153b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"
Sep 17 14:29:13.169112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.169083    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2efcf2e-8f5c-4457-b048-f9022f153b2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9\" (UID: \"f2efcf2e-8f5c-4457-b048-f9022f153b2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"
Sep 17 14:29:13.358757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.358712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"
Sep 17 14:29:13.460407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.459771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:13.675488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.675025    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"]
Sep 17 14:29:13.679019 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:29:13.678970    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2efcf2e_8f5c_4457_b048_f9022f153b2a.slice/crio-3bfbd0adc0f29aef31dbc09ebe1bf656de2415173118168e39ac0da697d62f11 WatchSource:0}: Error finding container 3bfbd0adc0f29aef31dbc09ebe1bf656de2415173118168e39ac0da697d62f11: Status 404 returned error can't find the container with id 3bfbd0adc0f29aef31dbc09ebe1bf656de2415173118168e39ac0da697d62f11
Sep 17 14:29:13.712784 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:13.712746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9" event={"ID":"f2efcf2e-8f5c-4457-b048-f9022f153b2a","Type":"ContainerStarted","Data":"3bfbd0adc0f29aef31dbc09ebe1bf656de2415173118168e39ac0da697d62f11"}
Sep 17 14:29:14.574280 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:14.574225    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:14.720203 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:14.720172    2565 generic.go:356] "Generic (PLEG): container finished" podID="f2efcf2e-8f5c-4457-b048-f9022f153b2a" containerID="d8435814e8889d18809fef93d77d4cc9235d820798230d3c8cdd23bb6e4265c5" exitCode=0
Sep 17 14:29:14.720596 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:14.720280    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9" event={"ID":"f2efcf2e-8f5c-4457-b048-f9022f153b2a","Type":"ContainerDied","Data":"d8435814e8889d18809fef93d77d4cc9235d820798230d3c8cdd23bb6e4265c5"}
Sep 17 14:29:15.922853 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:15.922826    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"
Sep 17 14:29:16.010915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:16.010808    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2efcf2e-8f5c-4457-b048-f9022f153b2a-maas-api-service-ca\") pod \"f2efcf2e-8f5c-4457-b048-f9022f153b2a\" (UID: \"f2efcf2e-8f5c-4457-b048-f9022f153b2a\") "
Sep 17 14:29:16.011172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:16.011146    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2efcf2e-8f5c-4457-b048-f9022f153b2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2efcf2e-8f5c-4457-b048-f9022f153b2a" (UID: "f2efcf2e-8f5c-4457-b048-f9022f153b2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:29:16.112621 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:16.112535    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2efcf2e-8f5c-4457-b048-f9022f153b2a-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:16.485141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:16.485104    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:16.733424 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:16.733391    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"
Sep 17 14:29:16.733612 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:16.733397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9" event={"ID":"f2efcf2e-8f5c-4457-b048-f9022f153b2a","Type":"ContainerDied","Data":"3bfbd0adc0f29aef31dbc09ebe1bf656de2415173118168e39ac0da697d62f11"}
Sep 17 14:29:16.733612 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:16.733505    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bfbd0adc0f29aef31dbc09ebe1bf656de2415173118168e39ac0da697d62f11"
Sep 17 14:29:17.061323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.061155    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:17.745137 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.744857    2565 generic.go:356] "Generic (PLEG): container finished" podID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerID="0ce4d3c8f8846469189d32a8fade8e0b0073a02257be9e70622ee62fbd974bde" exitCode=0
Sep 17 14:29:17.745137 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.744965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" event={"ID":"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb","Type":"ContainerDied","Data":"0ce4d3c8f8846469189d32a8fade8e0b0073a02257be9e70622ee62fbd974bde"}
Sep 17 14:29:17.745137 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.745023    2565 scope.go:117] "RemoveContainer" containerID="0e667d70396575a36ef701bb8c71517c551677509ab9716198d3f002949e2c16"
Sep 17 14:29:17.892504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.892468    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:29:17.934911 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.934871    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-token\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.934964    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-podinfo\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-data\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935047    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-socket\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935074    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69jfn\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-kube-api-access-69jfn\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935337 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-certs\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935373    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-data" (OuterVolumeSpecName: "istio-data") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:17.935632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935416    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:17.935975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:17.935975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935628    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-credential-socket\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935689    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istiod-ca-cert\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935745    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-envoy\") pod \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\" (UID: \"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb\") "
Sep 17 14:29:17.935975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.935853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:17.936416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.936161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:29:17.936416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.936369    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:17.936416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.936390    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:17.936416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.936406    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:17.936656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.936420    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:17.936656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.936433    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:17.937510 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.937484    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:29:17.937608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.937528    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-token" (OuterVolumeSpecName: "istio-token") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:29:17.937841 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.937822    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-kube-api-access-69jfn" (OuterVolumeSpecName: "kube-api-access-69jfn") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "kube-api-access-69jfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:29:17.938358 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.938339    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" (UID: "befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:17.957018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:17.956825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:18.037048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.037012    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69jfn\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-kube-api-access-69jfn\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:18.037048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.037044    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:18.037048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.037055    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:18.037354 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.037063    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:18.608860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.608822    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:18.753821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.753782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs" event={"ID":"befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb","Type":"ContainerDied","Data":"b6408e1a8fecef314bdfc11f5f7fea654d0bb220043e3032112d65fbeb9078fa"}
Sep 17 14:29:18.754007 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.753859    2565 scope.go:117] "RemoveContainer" containerID="0ce4d3c8f8846469189d32a8fade8e0b0073a02257be9e70622ee62fbd974bde"
Sep 17 14:29:18.754007 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.753868    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"
Sep 17 14:29:18.773993 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.773961    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:18.793979 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.793936    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"]
Sep 17 14:29:18.804528 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.804496    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2e1b82-openshift-default-5f94d9fd8b-pvsvs"]
Sep 17 14:29:18.940730 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.940630    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"]
Sep 17 14:29:18.941207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941184    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy"
Sep 17 14:29:18.941207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941208    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy"
Sep 17 14:29:18.941400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941231    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2efcf2e-8f5c-4457-b048-f9022f153b2a" containerName="revoke-keys"
Sep 17 14:29:18.941400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941240    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2efcf2e-8f5c-4457-b048-f9022f153b2a" containerName="revoke-keys"
Sep 17 14:29:18.941400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941268    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy"
Sep 17 14:29:18.941400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941276    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy"
Sep 17 14:29:18.941591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941410    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy"
Sep 17 14:29:18.941591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941426    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2efcf2e-8f5c-4457-b048-f9022f153b2a" containerName="revoke-keys"
Sep 17 14:29:18.941591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.941438    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" containerName="istio-proxy"
Sep 17 14:29:18.946499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.946465    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:18.950157 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.949395    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d2d17aff-openshift-default-dockercfg-n8p87\""
Sep 17 14:29:18.966652 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:18.966615    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"]
Sep 17 14:29:19.011484 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.011452    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2b461-0299-4edf-a657-fb8e646c6398-istiod-ca-cert\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-socket\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047472    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-podinfo\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-token\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-data\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8bnl\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-kube-api-access-v8bnl\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047620    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-certs\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047663    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-envoy\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.048135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.047692    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-credential-socket\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.127534 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.127490    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:19.149397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2b461-0299-4edf-a657-fb8e646c6398-istiod-ca-cert\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.149614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149484    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-socket\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.149614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149522    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-podinfo\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.149614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-token\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.149614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-data\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.149847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8bnl\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-kube-api-access-v8bnl\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.149847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149683    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-certs\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.149847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149725    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-envoy\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.149847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-credential-socket\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.150073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.149917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-socket\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.150174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.150147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-credential-socket\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.150276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.150169    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-data\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.150434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.150414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-certs\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.151549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.151520    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2b461-0299-4edf-a657-fb8e646c6398-istiod-ca-cert\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.153277 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.153209    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-envoy\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.154103 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.154076    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-podinfo\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.159902 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.159878    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8bnl\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-kube-api-access-v8bnl\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.159902 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.159896    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-token\") pod \"e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") " pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.235652 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.235550    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:19.261354 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.261318    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:19.522416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.522337    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb" path="/var/lib/kubelet/pods/befd2c68-2119-4c0d-bcdc-0a7b2bd68fdb/volumes"
Sep 17 14:29:19.577501 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:29:19.577456    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa2b461_0299_4edf_a657_fb8e646c6398.slice/crio-e6c0af248e59bd44cffb56c40305793fc8d235de9703e8914d25c0d78d3f980b WatchSource:0}: Error finding container e6c0af248e59bd44cffb56c40305793fc8d235de9703e8914d25c0d78d3f980b: Status 404 returned error can't find the container with id e6c0af248e59bd44cffb56c40305793fc8d235de9703e8914d25c0d78d3f980b
Sep 17 14:29:19.577747 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.577721    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"]
Sep 17 14:29:19.580193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.580159    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:19.580323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.580239    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:19.580323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.580301    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:19.762488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.762439    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" event={"ID":"3fa2b461-0299-4edf-a657-fb8e646c6398","Type":"ContainerStarted","Data":"d95768977be315cee6fe711bdf2ac96266e341e880587a3478cdb1d506db9c2b"}
Sep 17 14:29:19.763026 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.762517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" event={"ID":"3fa2b461-0299-4edf-a657-fb8e646c6398","Type":"ContainerStarted","Data":"e6c0af248e59bd44cffb56c40305793fc8d235de9703e8914d25c0d78d3f980b"}
Sep 17 14:29:19.807671 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:19.807538    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" podStartSLOduration=1.8075137639999999 podStartE2EDuration="1.807513764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:19.802354373 +0000 UTC m=+1346.735859566" watchObservedRunningTime="2026-09-17 14:29:19.807513764 +0000 UTC m=+1346.741018957"
Sep 17 14:29:20.262465 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:20.262419    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:20.268311 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:20.268282    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:20.331169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:20.331132    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:20.773766 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:20.773731    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:20.774886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:20.774860    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:21.989638 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:21.989599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:22.158569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:22.158532    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:22.454441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:22.454397    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:22.790772 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:22.790681    2565 generic.go:356] "Generic (PLEG): container finished" podID="71952167-bbbe-4528-a9e1-5c79de58326b" containerID="8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1" exitCode=0
Sep 17 14:29:22.790772 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:22.790755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" event={"ID":"71952167-bbbe-4528-a9e1-5c79de58326b","Type":"ContainerDied","Data":"8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1"}
Sep 17 14:29:22.791292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:22.791248    2565 scope.go:117] "RemoveContainer" containerID="8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1"
Sep 17 14:29:22.792426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:22.792390    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:22.792515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:22.792461    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:22.792515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:22.792496    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:23.655073 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:23.655012    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:23.798450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:23.798417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" event={"ID":"71952167-bbbe-4528-a9e1-5c79de58326b","Type":"ContainerStarted","Data":"d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c"}
Sep 17 14:29:23.820890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:23.820860    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:29:23.821084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:23.821027    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 17 14:29:23.821084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:23.821071    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 17 14:29:24.638611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:24.638580    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:24.822108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:24.822074    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 17 14:29:24.822511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:24.822134    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 17 14:29:25.475990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:25.475952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:25.591214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:25.591185    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:25.822116 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:25.822027    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 17 14:29:25.822116 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:25.822100    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 17 14:29:26.434279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:26.434232    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:26.821426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:26.821340    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 17 14:29:26.821426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:26.821392    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 17 14:29:27.645662 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:27.645620    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:27.821317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:27.821284    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 17 14:29:27.821494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:27.821368    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 17 14:29:28.781874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:28.781838    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:28.823619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:28.823579    2565 patch_prober.go:28] interesting pod/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Sep 17 14:29:28.823930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:28.823892    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Sep 17 14:29:28.921613 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:28.921569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:29.081383 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:29.081348    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:29.779488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:29.779442    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:29.825738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:29.825711    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:29:29.826127 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:29.826071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:29:29.826660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:29.826641    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:29:29.901647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:29.901604    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:30.059324 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.059210    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"]
Sep 17 14:29:30.064159 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.064126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.067591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.067545    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d2d17aff-dockercfg-cskwj\""
Sep 17 14:29:30.068087 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.067980    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d2d17aff\""
Sep 17 14:29:30.085274 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.084531    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"]
Sep 17 14:29:30.188166 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.188134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sqk5\" (UniqueName: \"kubernetes.io/projected/69cc0e3b-a67a-42d2-bb00-42b9175513d4-kube-api-access-5sqk5\") pod \"payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh\" (UID: \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\") " pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.188397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.188174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cc0e3b-a67a-42d2-bb00-42b9175513d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh\" (UID: \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\") " pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.204405 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.204378    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:30.288982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.288950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sqk5\" (UniqueName: \"kubernetes.io/projected/69cc0e3b-a67a-42d2-bb00-42b9175513d4-kube-api-access-5sqk5\") pod \"payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh\" (UID: \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\") " pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.289143 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.289008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cc0e3b-a67a-42d2-bb00-42b9175513d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh\" (UID: \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\") " pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.289213 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.289162    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:30.289751 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.289734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cc0e3b-a67a-42d2-bb00-42b9175513d4-plugins-config-volume\") pod \"payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh\" (UID: \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\") " pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.301804 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.301778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sqk5\" (UniqueName: \"kubernetes.io/projected/69cc0e3b-a67a-42d2-bb00-42b9175513d4-kube-api-access-5sqk5\") pod \"payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh\" (UID: \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\") " pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.384031 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.383991    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:30.384919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.384883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.481627 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.481506    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:30.638381 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.638302    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"]
Sep 17 14:29:30.640652 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.640623    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-28f011-ptcc9"]
Sep 17 14:29:30.642531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.642507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:30.660878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.660855    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"]
Sep 17 14:29:30.662797 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:29:30.662771    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cc0e3b_a67a_42d2_bb00_42b9175513d4.slice/crio-dda07490a846da2b30e02ab3125df636dbef30dc1f33daf812ff2c7856c85bbe WatchSource:0}: Error finding container dda07490a846da2b30e02ab3125df636dbef30dc1f33daf812ff2c7856c85bbe: Status 404 returned error can't find the container with id dda07490a846da2b30e02ab3125df636dbef30dc1f33daf812ff2c7856c85bbe
Sep 17 14:29:30.855363 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.855325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh" event={"ID":"69cc0e3b-a67a-42d2-bb00-42b9175513d4","Type":"ContainerStarted","Data":"f7a0d3437e7b7fa6ebb098566eb2037b8a7c677313bdc827e124b1e7893c0f20"}
Sep 17 14:29:30.855835 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.855386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh" event={"ID":"69cc0e3b-a67a-42d2-bb00-42b9175513d4","Type":"ContainerStarted","Data":"dda07490a846da2b30e02ab3125df636dbef30dc1f33daf812ff2c7856c85bbe"}
Sep 17 14:29:30.855835 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.855435    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:30.894479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:30.894374    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh" podStartSLOduration=0.89435381 podStartE2EDuration="894.35381ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:30.89393715 +0000 UTC m=+1357.827442343" watchObservedRunningTime="2026-09-17 14:29:30.89435381 +0000 UTC m=+1357.827859003"
Sep 17 14:29:31.015668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:31.015587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:31.485689 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:31.485650    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:31.521463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:31.521430    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2efcf2e-8f5c-4457-b048-f9022f153b2a" path="/var/lib/kubelet/pods/f2efcf2e-8f5c-4457-b048-f9022f153b2a/volumes"
Sep 17 14:29:32.436017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:32.435980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:33.358427 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:33.358396    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:34.260519 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:34.260490    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:35.055022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:35.054984    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:36.080888 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:36.080854    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:36.218902 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:36.218873    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:37.959305 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:37.959245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:38.058550 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:38.058517    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:38.222473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:38.222383    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"]
Sep 17 14:29:38.222656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:38.222633    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy" containerID="cri-o://998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17" gracePeriod=30
Sep 17 14:29:38.236783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:38.236754    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:38.776546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:38.776511    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:39.135147 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:39.135108    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:39.667195 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:39.667160    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:42.868932 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:42.868900    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:29:43.484263 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.484237    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:29:43.550429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550343    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-socket\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.550589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550450    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlrrq\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-kube-api-access-dlrrq\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.550589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550539    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-envoy\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.550589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550565    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-credential-socket\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.550755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550622    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-data\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.550755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550653    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-token\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.550755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550681    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab092f48-a46e-498d-9f3c-369cc313e5c5-istiod-ca-cert\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.550755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550693    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:43.550755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550711    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-podinfo\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.551018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-certs\") pod \"ab092f48-a46e-498d-9f3c-369cc313e5c5\" (UID: \"ab092f48-a46e-498d-9f3c-369cc313e5c5\") "
Sep 17 14:29:43.551018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.550956    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:43.551893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.551167    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:43.551893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.551221    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.551893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.551266    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.551893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.551468    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab092f48-a46e-498d-9f3c-369cc313e5c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:29:43.551893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.551858    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:43.555282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.554365    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:29:43.555282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.554669    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:29:43.555282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.554755    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:29:43.555637 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.555621    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-kube-api-access-dlrrq" (OuterVolumeSpecName: "kube-api-access-dlrrq") pod "ab092f48-a46e-498d-9f3c-369cc313e5c5" (UID: "ab092f48-a46e-498d-9f3c-369cc313e5c5"). InnerVolumeSpecName "kube-api-access-dlrrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:29:43.652447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.652422    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab092f48-a46e-498d-9f3c-369cc313e5c5-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.652447 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.652449    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.652624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.652464    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.652624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.652478    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrrq\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-kube-api-access-dlrrq\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.652624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.652490    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.652624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.652503    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.652624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.652514    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab092f48-a46e-498d-9f3c-369cc313e5c5-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:29:43.941013 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.940973    2565 generic.go:356] "Generic (PLEG): container finished" podID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerID="998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17" exitCode=0
Sep 17 14:29:43.941439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.941019    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"
Sep 17 14:29:43.941439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.941064    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" event={"ID":"ab092f48-a46e-498d-9f3c-369cc313e5c5","Type":"ContainerDied","Data":"998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17"}
Sep 17 14:29:43.941439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.941119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q" event={"ID":"ab092f48-a46e-498d-9f3c-369cc313e5c5","Type":"ContainerDied","Data":"ff6329bda59355793b007d4e8b079e02212fab57825b1b0007550ba1b9d6560a"}
Sep 17 14:29:43.941439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.941145    2565 scope.go:117] "RemoveContainer" containerID="998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17"
Sep 17 14:29:43.944503 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.944470    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:43.952337 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.952317    2565 scope.go:117] "RemoveContainer" containerID="26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01"
Sep 17 14:29:43.961745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.961726    2565 scope.go:117] "RemoveContainer" containerID="998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17"
Sep 17 14:29:43.962040 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:29:43.962021    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17\": container with ID starting with 998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17 not found: ID does not exist" containerID="998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17"
Sep 17 14:29:43.962088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.962051    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17"} err="failed to get container status \"998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17\": rpc error: code = NotFound desc = could not find container \"998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17\": container with ID starting with 998d601f231fd142f292c0b819474d618a63ec7e560e8a573abb8c6bb099ad17 not found: ID does not exist"
Sep 17 14:29:43.962088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.962069    2565 scope.go:117] "RemoveContainer" containerID="26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01"
Sep 17 14:29:43.962353 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:29:43.962335    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01\": container with ID starting with 26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01 not found: ID does not exist" containerID="26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01"
Sep 17 14:29:43.962404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.962360    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01"} err="failed to get container status \"26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01\": rpc error: code = NotFound desc = could not find container \"26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01\": container with ID starting with 26924bc2f95b25fde48656d1487d8ad5c0bd37f7033e77f75d2d65945d302d01 not found: ID does not exist"
Sep 17 14:29:43.966507 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.966486    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"]
Sep 17 14:29:43.975946 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:43.975914    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-28f011-openshift-default-65dbc584cb-fgr9q"]
Sep 17 14:29:44.658155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:44.658118    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:44.760813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:44.760780    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:45.167758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:45.167725    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:45.520366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:45.520246    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" path="/var/lib/kubelet/pods/ab092f48-a46e-498d-9f3c-369cc313e5c5/volumes"
Sep 17 14:29:46.054226 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:46.054185    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:47.422972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:47.422936    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:48.689062 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:48.689026    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:49.692483 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:49.692450    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:50.637544 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:50.637515    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:50.741098 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:50.741064    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:50.836492 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:50.836453    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:51.176068 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.176033    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:51.310183 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.310145    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:51.393914 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.393877    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"]
Sep 17 14:29:51.394433 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.394414    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy"
Sep 17 14:29:51.394504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.394435    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy"
Sep 17 14:29:51.394504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.394446    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy"
Sep 17 14:29:51.394504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.394453    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy"
Sep 17 14:29:51.394602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.394536    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy"
Sep 17 14:29:51.394602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.394544    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab092f48-a46e-498d-9f3c-369cc313e5c5" containerName="istio-proxy"
Sep 17 14:29:51.399540 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.399523    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:51.404236 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.404207    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3da4be-dockercfg-rmp9z\""
Sep 17 14:29:51.405245 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.405220    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3da4be\""
Sep 17 14:29:51.407291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.407266    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"]
Sep 17 14:29:51.540033 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.539873    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv62f\" (UniqueName: \"kubernetes.io/projected/71b552e9-457f-4a56-8ce0-182c41a78d17-kube-api-access-xv62f\") pod \"payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7\" (UID: \"71b552e9-457f-4a56-8ce0-182c41a78d17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:51.540223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.540134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b552e9-457f-4a56-8ce0-182c41a78d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7\" (UID: \"71b552e9-457f-4a56-8ce0-182c41a78d17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:51.641723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.641675    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv62f\" (UniqueName: \"kubernetes.io/projected/71b552e9-457f-4a56-8ce0-182c41a78d17-kube-api-access-xv62f\") pod \"payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7\" (UID: \"71b552e9-457f-4a56-8ce0-182c41a78d17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:51.641907 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.641792    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b552e9-457f-4a56-8ce0-182c41a78d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7\" (UID: \"71b552e9-457f-4a56-8ce0-182c41a78d17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:51.642602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.642577    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b552e9-457f-4a56-8ce0-182c41a78d17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7\" (UID: \"71b552e9-457f-4a56-8ce0-182c41a78d17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:51.652331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.652298    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv62f\" (UniqueName: \"kubernetes.io/projected/71b552e9-457f-4a56-8ce0-182c41a78d17-kube-api-access-xv62f\") pod \"payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7\" (UID: \"71b552e9-457f-4a56-8ce0-182c41a78d17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:51.712450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.712409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:51.787281 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.787238    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:51.888353 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:51.888280    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:52.021646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:52.021615    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"]
Sep 17 14:29:52.023953 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:29:52.023923    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b552e9_457f_4a56_8ce0_182c41a78d17.slice/crio-e6862057bbc70f5aa2abe546f746a9fbe921e9b67060cf58962884af4f127050 WatchSource:0}: Error finding container e6862057bbc70f5aa2abe546f746a9fbe921e9b67060cf58962884af4f127050: Status 404 returned error can't find the container with id e6862057bbc70f5aa2abe546f746a9fbe921e9b67060cf58962884af4f127050
Sep 17 14:29:52.656959 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:52.656889    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:53.004487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:53.004383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7" event={"ID":"71b552e9-457f-4a56-8ce0-182c41a78d17","Type":"ContainerStarted","Data":"f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42"}
Sep 17 14:29:53.004487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:53.004456    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:29:53.004487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:53.004480    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7" event={"ID":"71b552e9-457f-4a56-8ce0-182c41a78d17","Type":"ContainerStarted","Data":"e6862057bbc70f5aa2abe546f746a9fbe921e9b67060cf58962884af4f127050"}
Sep 17 14:29:53.033635 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:53.033575    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7" podStartSLOduration=2.033559287 podStartE2EDuration="2.033559287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:29:53.031505559 +0000 UTC m=+1379.965010752" watchObservedRunningTime="2026-09-17 14:29:53.033559287 +0000 UTC m=+1379.967064479"
Sep 17 14:29:53.569975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:53.569844    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:53.717462 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:53.717430    2565 scope.go:117] "RemoveContainer" containerID="d2889e117e1e4a4432385780aff3d5c89a77c011fc8d9ffbb995c0990d46fc69"
Sep 17 14:29:54.362028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:54.361946    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:55.297589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:55.297546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:56.200317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:56.200285    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:29:58.039049 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:58.039016    2565 generic.go:356] "Generic (PLEG): container finished" podID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerID="d95768977be315cee6fe711bdf2ac96266e341e880587a3478cdb1d506db9c2b" exitCode=0
Sep 17 14:29:58.039629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:58.039081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" event={"ID":"3fa2b461-0299-4edf-a657-fb8e646c6398","Type":"ContainerDied","Data":"d95768977be315cee6fe711bdf2ac96266e341e880587a3478cdb1d506db9c2b"}
Sep 17 14:29:58.039719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:58.039695    2565 scope.go:117] "RemoveContainer" containerID="d95768977be315cee6fe711bdf2ac96266e341e880587a3478cdb1d506db9c2b"
Sep 17 14:29:58.041054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:58.041025    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:58.041124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:58.041096    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:58.041162 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:58.041131    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:29:59.046764 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:59.046722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" event={"ID":"3fa2b461-0299-4edf-a657-fb8e646c6398","Type":"ContainerStarted","Data":"b1803658ad3f58c0e99f758c3e2e41d4ff50add930f2bdd10c2cf760b66bcc6e"}
Sep 17 14:29:59.261976 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:59.261937    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:29:59.262414 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:59.262303    2565 patch_prober.go:28] interesting pod/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 17 14:29:59.262653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:29:59.262624    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 17 14:30:00.208978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.208941    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"]
Sep 17 14:30:00.211708 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.211680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"
Sep 17 14:30:00.214698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.214677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:30:00.222514 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.222494    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"
Sep 17 14:30:00.233178 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.233025    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"]
Sep 17 14:30:00.236100 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.236067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"]
Sep 17 14:30:00.241099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.240137    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf"]
Sep 17 14:30:00.241099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.240231    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"
Sep 17 14:30:00.244350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.244297    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"]
Sep 17 14:30:00.244522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.244403    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf"
Sep 17 14:30:00.257276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.247838    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"
Sep 17 14:30:00.270929 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.270895    2565 patch_prober.go:28] interesting pod/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 17 14:30:00.271205 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.271174    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 17 14:30:00.297310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.282945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"]
Sep 17 14:30:00.298975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.298932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf"]
Sep 17 14:30:00.309182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.309158    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"]
Sep 17 14:30:00.313143 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.313118    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"
Sep 17 14:30:00.324494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.323717    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf"
Sep 17 14:30:00.418750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.418635    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"
Sep 17 14:30:00.575054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.575010    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"]
Sep 17 14:30:00.582952 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:30:00.582913    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63110548_178d_4f6d_a2a5_68f3a4aa7f8c.slice/crio-935a8b8650bd72635be524c2ab4a86b4031cdece9446d1f18de230866d7797f2 WatchSource:0}: Error finding container 935a8b8650bd72635be524c2ab4a86b4031cdece9446d1f18de230866d7797f2: Status 404 returned error can't find the container with id 935a8b8650bd72635be524c2ab4a86b4031cdece9446d1f18de230866d7797f2
Sep 17 14:30:00.646590 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.646559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"]
Sep 17 14:30:00.647738 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:30:00.647711    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e22d522_f668_4515_81c9_6596acbcf390.slice/crio-993a6f1005203bd1974a7db3f6cd081c373c64b623c69b6e6309c1d5099fbef2 WatchSource:0}: Error finding container 993a6f1005203bd1974a7db3f6cd081c373c64b623c69b6e6309c1d5099fbef2: Status 404 returned error can't find the container with id 993a6f1005203bd1974a7db3f6cd081c373c64b623c69b6e6309c1d5099fbef2
Sep 17 14:30:00.700216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.700189    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf"]
Sep 17 14:30:00.710542 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:30:00.710516    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd315bd8b_df1e_44be_8742_5c0ee0ba9261.slice/crio-e61fe54db9b4e9e6fc0681c67cae093b93d5f44f1b8175de35503d2fa741df11 WatchSource:0}: Error finding container e61fe54db9b4e9e6fc0681c67cae093b93d5f44f1b8175de35503d2fa741df11: Status 404 returned error can't find the container with id e61fe54db9b4e9e6fc0681c67cae093b93d5f44f1b8175de35503d2fa741df11
Sep 17 14:30:00.777000 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:00.776956    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"]
Sep 17 14:30:00.784543 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:30:00.784514    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod783fe4d4_3c4a_41fd_ad3d_2b01090dca0f.slice/crio-d7dd225d71a1fc4981337b037cb673b9c82d10be59b6c92e49751634a18cb333 WatchSource:0}: Error finding container d7dd225d71a1fc4981337b037cb673b9c82d10be59b6c92e49751634a18cb333: Status 404 returned error can't find the container with id d7dd225d71a1fc4981337b037cb673b9c82d10be59b6c92e49751634a18cb333
Sep 17 14:30:01.080278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.078936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw" event={"ID":"63110548-178d-4f6d-a2a5-68f3a4aa7f8c","Type":"ContainerStarted","Data":"1b904704956ca85948330a5c8854ccd55f67d38caf2bf686c5d5a9e4ca045e8b"}
Sep 17 14:30:01.080278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.079006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw" event={"ID":"63110548-178d-4f6d-a2a5-68f3a4aa7f8c","Type":"ContainerStarted","Data":"935a8b8650bd72635be524c2ab4a86b4031cdece9446d1f18de230866d7797f2"}
Sep 17 14:30:01.084390 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.083890    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p" event={"ID":"4e22d522-f668-4515-81c9-6596acbcf390","Type":"ContainerStarted","Data":"993a6f1005203bd1974a7db3f6cd081c373c64b623c69b6e6309c1d5099fbef2"}
Sep 17 14:30:01.086400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.086375    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf" event={"ID":"d315bd8b-df1e-44be-8742-5c0ee0ba9261","Type":"ContainerStarted","Data":"e61fe54db9b4e9e6fc0681c67cae093b93d5f44f1b8175de35503d2fa741df11"}
Sep 17 14:30:01.088169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.088146    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7" event={"ID":"783fe4d4-3c4a-41fd-ad3d-2b01090dca0f","Type":"ContainerStarted","Data":"d7dd225d71a1fc4981337b037cb673b9c82d10be59b6c92e49751634a18cb333"}
Sep 17 14:30:01.097816 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.097748    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw" podStartSLOduration=1.09773245 podStartE2EDuration="1.09773245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:01.096057046 +0000 UTC m=+1388.029562239" watchObservedRunningTime="2026-09-17 14:30:01.09773245 +0000 UTC m=+1388.031237642"
Sep 17 14:30:01.115188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.115128    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf" podStartSLOduration=1.115104551 podStartE2EDuration="1.115104551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:01.112243868 +0000 UTC m=+1388.045749061" watchObservedRunningTime="2026-09-17 14:30:01.115104551 +0000 UTC m=+1388.048609746"
Sep 17 14:30:01.263760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.263197    2565 patch_prober.go:28] interesting pod/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 17 14:30:01.263760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:01.263280    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 17 14:30:02.096228 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.096190    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p" event={"ID":"4e22d522-f668-4515-81c9-6596acbcf390","Type":"ContainerStarted","Data":"8a65805b853bbc176203676b2bb3eb24f3d49abf07742ce1b2f772804b6f801e"}
Sep 17 14:30:02.097918 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.097897    2565 generic.go:356] "Generic (PLEG): container finished" podID="d315bd8b-df1e-44be-8742-5c0ee0ba9261" containerID="8a7cbffb2eb8698bc2336035bfeb73854af748308fd424fea0c6073d8a4f71e3" exitCode=0
Sep 17 14:30:02.098102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.098087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf" event={"ID":"d315bd8b-df1e-44be-8742-5c0ee0ba9261","Type":"ContainerDied","Data":"8a7cbffb2eb8698bc2336035bfeb73854af748308fd424fea0c6073d8a4f71e3"}
Sep 17 14:30:02.099672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.099649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7" event={"ID":"783fe4d4-3c4a-41fd-ad3d-2b01090dca0f","Type":"ContainerStarted","Data":"df131e6a30a2ce4f9ac14e0463c283504468e795d9934d8b77c79698b34efac4"}
Sep 17 14:30:02.101489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.101470    2565 generic.go:356] "Generic (PLEG): container finished" podID="63110548-178d-4f6d-a2a5-68f3a4aa7f8c" containerID="1b904704956ca85948330a5c8854ccd55f67d38caf2bf686c5d5a9e4ca045e8b" exitCode=0
Sep 17 14:30:02.101583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.101515    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw" event={"ID":"63110548-178d-4f6d-a2a5-68f3a4aa7f8c","Type":"ContainerDied","Data":"1b904704956ca85948330a5c8854ccd55f67d38caf2bf686c5d5a9e4ca045e8b"}
Sep 17 14:30:02.128275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.128187    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p" podStartSLOduration=2.128172962 podStartE2EDuration="2.128172962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:02.125803218 +0000 UTC m=+1389.059308409" watchObservedRunningTime="2026-09-17 14:30:02.128172962 +0000 UTC m=+1389.061678160"
Sep 17 14:30:02.204113 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.204053    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7" podStartSLOduration=2.204037084 podStartE2EDuration="2.204037084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:02.180494475 +0000 UTC m=+1389.113999667" watchObservedRunningTime="2026-09-17 14:30:02.204037084 +0000 UTC m=+1389.137542276"
Sep 17 14:30:02.262327 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.262280    2565 patch_prober.go:28] interesting pod/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 17 14:30:02.262545 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:02.262356    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 17 14:30:03.109537 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:03.109499    2565 generic.go:356] "Generic (PLEG): container finished" podID="783fe4d4-3c4a-41fd-ad3d-2b01090dca0f" containerID="df131e6a30a2ce4f9ac14e0463c283504468e795d9934d8b77c79698b34efac4" exitCode=0
Sep 17 14:30:03.109980 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:03.109631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7" event={"ID":"783fe4d4-3c4a-41fd-ad3d-2b01090dca0f","Type":"ContainerDied","Data":"df131e6a30a2ce4f9ac14e0463c283504468e795d9934d8b77c79698b34efac4"}
Sep 17 14:30:03.111598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:03.111574    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e22d522-f668-4515-81c9-6596acbcf390" containerID="8a65805b853bbc176203676b2bb3eb24f3d49abf07742ce1b2f772804b6f801e" exitCode=0
Sep 17 14:30:03.111857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:03.111841    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p" event={"ID":"4e22d522-f668-4515-81c9-6596acbcf390","Type":"ContainerDied","Data":"8a65805b853bbc176203676b2bb3eb24f3d49abf07742ce1b2f772804b6f801e"}
Sep 17 14:30:03.263675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:03.263018    2565 patch_prober.go:28] interesting pod/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Sep 17 14:30:03.263675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:03.263111    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Sep 17 14:30:03.376353 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:03.373466    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"
Sep 17 14:30:03.385279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:03.381442    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf"
Sep 17 14:30:04.013411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.013375    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:30:04.120744 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.120712    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"
Sep 17 14:30:04.120744 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.120722    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw" event={"ID":"63110548-178d-4f6d-a2a5-68f3a4aa7f8c","Type":"ContainerDied","Data":"935a8b8650bd72635be524c2ab4a86b4031cdece9446d1f18de230866d7797f2"}
Sep 17 14:30:04.121230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.120776    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="935a8b8650bd72635be524c2ab4a86b4031cdece9446d1f18de230866d7797f2"
Sep 17 14:30:04.122444 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.122414    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf"
Sep 17 14:30:04.122593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.122461    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827590-85dxf" event={"ID":"d315bd8b-df1e-44be-8742-5c0ee0ba9261","Type":"ContainerDied","Data":"e61fe54db9b4e9e6fc0681c67cae093b93d5f44f1b8175de35503d2fa741df11"}
Sep 17 14:30:04.122593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.122500    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e61fe54db9b4e9e6fc0681c67cae093b93d5f44f1b8175de35503d2fa741df11"
Sep 17 14:30:04.267099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.267015    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:30:04.267694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.267670    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:30:04.268606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.268578    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:30:04.308506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.308480    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"
Sep 17 14:30:04.309558 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:04.309540    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"
Sep 17 14:30:05.130438 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:05.130406    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"
Sep 17 14:30:05.130912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:05.130436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p" event={"ID":"4e22d522-f668-4515-81c9-6596acbcf390","Type":"ContainerDied","Data":"993a6f1005203bd1974a7db3f6cd081c373c64b623c69b6e6309c1d5099fbef2"}
Sep 17 14:30:05.130912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:05.130489    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="993a6f1005203bd1974a7db3f6cd081c373c64b623c69b6e6309c1d5099fbef2"
Sep 17 14:30:05.132005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:05.131983    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"
Sep 17 14:30:05.132150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:05.132091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7" event={"ID":"783fe4d4-3c4a-41fd-ad3d-2b01090dca0f","Type":"ContainerDied","Data":"d7dd225d71a1fc4981337b037cb673b9c82d10be59b6c92e49751634a18cb333"}
Sep 17 14:30:05.132150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:05.132130    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7dd225d71a1fc4981337b037cb673b9c82d10be59b6c92e49751634a18cb333"
Sep 17 14:30:13.053953 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:13.053906    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:13.963718 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:13.963647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:14.604742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:14.604679    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:14.939919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:14.939832    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:15.040649 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:15.040610    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:15.365799 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:15.365769    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:15.555247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:15.555189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:16.168620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:16.168578    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:16.988847 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:16.988810    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:18.235017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:18.234983    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:18.839316 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:18.839245    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:18.966287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:18.966216    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:19.342494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:19.342461    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:20.415799 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:20.415762    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:20.521398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:20.521361    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:20.669192 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:20.669107    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:21.364986 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:21.364950    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:21.934296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:21.934247    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:22.042953 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:22.042922    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:22.918925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:22.918893    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:23.810232 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:23.810201    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:24.662864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:24.662833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:25.648684 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:25.648652    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:26.701434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:26.701401    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:35.323744 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.323709    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"]
Sep 17 14:30:35.324416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324399    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63110548-178d-4f6d-a2a5-68f3a4aa7f8c" containerName="cleanup"
Sep 17 14:30:35.324477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324420    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="63110548-178d-4f6d-a2a5-68f3a4aa7f8c" containerName="cleanup"
Sep 17 14:30:35.324477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324441    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e22d522-f668-4515-81c9-6596acbcf390" containerName="cleanup"
Sep 17 14:30:35.324477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324449    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e22d522-f668-4515-81c9-6596acbcf390" containerName="cleanup"
Sep 17 14:30:35.324477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324473    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="783fe4d4-3c4a-41fd-ad3d-2b01090dca0f" containerName="cleanup"
Sep 17 14:30:35.324608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324482    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="783fe4d4-3c4a-41fd-ad3d-2b01090dca0f" containerName="cleanup"
Sep 17 14:30:35.324608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324494    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d315bd8b-df1e-44be-8742-5c0ee0ba9261" containerName="cleanup"
Sep 17 14:30:35.324608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324502    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d315bd8b-df1e-44be-8742-5c0ee0ba9261" containerName="cleanup"
Sep 17 14:30:35.324709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324615    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e22d522-f668-4515-81c9-6596acbcf390" containerName="cleanup"
Sep 17 14:30:35.324709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324629    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d315bd8b-df1e-44be-8742-5c0ee0ba9261" containerName="cleanup"
Sep 17 14:30:35.324709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324638    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="63110548-178d-4f6d-a2a5-68f3a4aa7f8c" containerName="cleanup"
Sep 17 14:30:35.324709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.324647    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="783fe4d4-3c4a-41fd-ad3d-2b01090dca0f" containerName="cleanup"
Sep 17 14:30:35.329719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.329694    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.334241 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.334101    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d2d17aff\"/\"default-dockercfg-njfqw\""
Sep 17 14:30:35.334241 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.334119    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d2d17aff\"/\"openshift-service-ca.crt\""
Sep 17 14:30:35.334241 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.334209    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d2d17aff\"/\"e2e-lifecycle-model-d2d17aff-kserve-self-signed-certs\""
Sep 17 14:30:35.334517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.334456    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d2d17aff\"/\"kube-root-ca.crt\""
Sep 17 14:30:35.341056 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.341030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"]
Sep 17 14:30:35.364614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.364577    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:35.478813 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.478781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-home\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.478972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.478846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6krs\" (UniqueName: \"kubernetes.io/projected/a11a370f-c8f1-4819-9311-838f877f13c2-kube-api-access-w6krs\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.478972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.478884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-model-cache\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.478972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.478938    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-tmp-dir\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.478972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.478961    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a370f-c8f1-4819-9311-838f877f13c2-tls-certs\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.479155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.479012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-dshm\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.580301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.580199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a370f-c8f1-4819-9311-838f877f13c2-tls-certs\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.580457 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.580335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-dshm\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.580563 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.580532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-home\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.580679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.580660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6krs\" (UniqueName: \"kubernetes.io/projected/a11a370f-c8f1-4819-9311-838f877f13c2-kube-api-access-w6krs\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.580770 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.580708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-model-cache\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.580875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.580850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-tmp-dir\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.580957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.580871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-home\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.581077 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.581051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-model-cache\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.581183 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.581159    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-tmp-dir\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.583097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.583078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-dshm\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.583301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.583283    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a370f-c8f1-4819-9311-838f877f13c2-tls-certs\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.596686 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.596659    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6krs\" (UniqueName: \"kubernetes.io/projected/a11a370f-c8f1-4819-9311-838f877f13c2-kube-api-access-w6krs\") pod \"e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") " pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.641455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.641428    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:35.932920 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:35.932896    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"]
Sep 17 14:30:35.935330 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:30:35.935296    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda11a370f_c8f1_4819_9311_838f877f13c2.slice/crio-625d86f88c10d535397aa6dd792956ad586d6d6775a7edb912f9a4f09837797b WatchSource:0}: Error finding container 625d86f88c10d535397aa6dd792956ad586d6d6775a7edb912f9a4f09837797b: Status 404 returned error can't find the container with id 625d86f88c10d535397aa6dd792956ad586d6d6775a7edb912f9a4f09837797b
Sep 17 14:30:36.329087 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:36.329040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2" event={"ID":"a11a370f-c8f1-4819-9311-838f877f13c2","Type":"ContainerStarted","Data":"179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f"}
Sep 17 14:30:36.329535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:36.329097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2" event={"ID":"a11a370f-c8f1-4819-9311-838f877f13c2","Type":"ContainerStarted","Data":"625d86f88c10d535397aa6dd792956ad586d6d6775a7edb912f9a4f09837797b"}
Sep 17 14:30:36.349450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:36.349404    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2" podStartSLOduration=1.349387944 podStartE2EDuration="1.349387944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:30:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:30:36.347672305 +0000 UTC m=+1423.281177499" watchObservedRunningTime="2026-09-17 14:30:36.349387944 +0000 UTC m=+1423.282893135"
Sep 17 14:30:38.239072 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:38.239037    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:43.362302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:43.362244    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:43.841199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:43.841164    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:44.109791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:44.109699    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:44.916901 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:44.916872    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:45.642532 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:45.642484    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:45.642532 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:45.642550    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:45.659346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:45.659315    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:46.288295 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:46.288197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:46.403309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:46.403263    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:30:47.188181 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:47.188143    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:48.322234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:48.322197    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:49.114697 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:49.114660    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:50.012400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:50.012368    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:30:53.773855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:53.773823    2565 scope.go:117] "RemoveContainer" containerID="d9e7b7db15cb4b56087433d1b3b8acbe1705cf7c84ed985c89f5053b6e5b928b"
Sep 17 14:30:53.782843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:53.782818    2565 scope.go:117] "RemoveContainer" containerID="b423fd85f357ca1a26e3e9c9f985d55e470c737fe1e5a827b4ea102c55a8f794"
Sep 17 14:30:58.328239 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:58.328201    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"]
Sep 17 14:30:58.353001 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:30:58.352964    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:12.818400 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:12.818360    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"]
Sep 17 14:31:12.823843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:12.823821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"
Sep 17 14:31:12.826661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:12.826636    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:31:12.826785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:12.826670    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:31:12.832211 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:12.831717    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"]
Sep 17 14:31:13.011507 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.011466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff41f195-e3b8-4886-8e6d-038f2e6c596c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl\" (UID: \"ff41f195-e3b8-4886-8e6d-038f2e6c596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"
Sep 17 14:31:13.113005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.112969    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff41f195-e3b8-4886-8e6d-038f2e6c596c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl\" (UID: \"ff41f195-e3b8-4886-8e6d-038f2e6c596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"
Sep 17 14:31:13.113666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.113647    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff41f195-e3b8-4886-8e6d-038f2e6c596c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl\" (UID: \"ff41f195-e3b8-4886-8e6d-038f2e6c596c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"
Sep 17 14:31:13.136051 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.136023    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"
Sep 17 14:31:13.136388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.136365    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:13.353644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.353457    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs" podUID="bc5096ed-0d69-4616-8f9f-3663dfbf940d" containerName="main" containerID="cri-o://134784e14dd5e181fcc1bb5f132596da9a3b30d83af2c0908f8fc862b66ef3dd" gracePeriod=45
Sep 17 14:31:13.418429 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.418396    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"]
Sep 17 14:31:13.427693 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:31:13.427661    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff41f195_e3b8_4886_8e6d_038f2e6c596c.slice/crio-d3e9da63e7bf0c1319d6b8f54306a52c779c0f836bdd468d056284e632590601 WatchSource:0}: Error finding container d3e9da63e7bf0c1319d6b8f54306a52c779c0f836bdd468d056284e632590601: Status 404 returned error can't find the container with id d3e9da63e7bf0c1319d6b8f54306a52c779c0f836bdd468d056284e632590601
Sep 17 14:31:13.561646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.561614    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc5096ed-0d69-4616-8f9f-3663dfbf940d" containerID="134784e14dd5e181fcc1bb5f132596da9a3b30d83af2c0908f8fc862b66ef3dd" exitCode=0
Sep 17 14:31:13.561807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.561687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs" event={"ID":"bc5096ed-0d69-4616-8f9f-3663dfbf940d","Type":"ContainerDied","Data":"134784e14dd5e181fcc1bb5f132596da9a3b30d83af2c0908f8fc862b66ef3dd"}
Sep 17 14:31:13.563273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.563121    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl" event={"ID":"ff41f195-e3b8-4886-8e6d-038f2e6c596c","Type":"ContainerStarted","Data":"d3e9da63e7bf0c1319d6b8f54306a52c779c0f836bdd468d056284e632590601"}
Sep 17 14:31:13.676063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.676037    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:31:13.821011 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.820985    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tls-certs\") pod \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") "
Sep 17 14:31:13.821407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821053    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-model-cache\") pod \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") "
Sep 17 14:31:13.821407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821172    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cpj4k\" (UniqueName: \"kubernetes.io/projected/bc5096ed-0d69-4616-8f9f-3663dfbf940d-kube-api-access-cpj4k\") pod \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") "
Sep 17 14:31:13.821407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821207    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tmp-dir\") pod \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") "
Sep 17 14:31:13.821407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821284    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-dshm\") pod \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") "
Sep 17 14:31:13.821407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821317    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-home\") pod \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\" (UID: \"bc5096ed-0d69-4616-8f9f-3663dfbf940d\") "
Sep 17 14:31:13.821407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821378    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc5096ed-0d69-4616-8f9f-3663dfbf940d" (UID: "bc5096ed-0d69-4616-8f9f-3663dfbf940d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:31:13.821689 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821473    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc5096ed-0d69-4616-8f9f-3663dfbf940d" (UID: "bc5096ed-0d69-4616-8f9f-3663dfbf940d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:31:13.821689 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821628    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-home" (OuterVolumeSpecName: "home") pod "bc5096ed-0d69-4616-8f9f-3663dfbf940d" (UID: "bc5096ed-0d69-4616-8f9f-3663dfbf940d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:31:13.821780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821762    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:13.821832 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821782    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:13.821832 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.821791    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:13.823234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.823213    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-dshm" (OuterVolumeSpecName: "dshm") pod "bc5096ed-0d69-4616-8f9f-3663dfbf940d" (UID: "bc5096ed-0d69-4616-8f9f-3663dfbf940d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:31:13.823547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.823525    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc5096ed-0d69-4616-8f9f-3663dfbf940d" (UID: "bc5096ed-0d69-4616-8f9f-3663dfbf940d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:13.823598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.823549    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5096ed-0d69-4616-8f9f-3663dfbf940d-kube-api-access-cpj4k" (OuterVolumeSpecName: "kube-api-access-cpj4k") pod "bc5096ed-0d69-4616-8f9f-3663dfbf940d" (UID: "bc5096ed-0d69-4616-8f9f-3663dfbf940d"). InnerVolumeSpecName "kube-api-access-cpj4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:13.922836 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.922779    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpj4k\" (UniqueName: \"kubernetes.io/projected/bc5096ed-0d69-4616-8f9f-3663dfbf940d-kube-api-access-cpj4k\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:13.922836 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.922800    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc5096ed-0d69-4616-8f9f-3663dfbf940d-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:13.922836 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:13.922810    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc5096ed-0d69-4616-8f9f-3663dfbf940d-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:14.281987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:14.281908    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:14.570822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:14.570740    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"
Sep 17 14:31:14.570822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:14.570756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs" event={"ID":"bc5096ed-0d69-4616-8f9f-3663dfbf940d","Type":"ContainerDied","Data":"04afdc5fca247b7c27ad88418204f1b7db101c574e6996c66b6f4396300ceb29"}
Sep 17 14:31:14.570822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:14.570805    2565 scope.go:117] "RemoveContainer" containerID="134784e14dd5e181fcc1bb5f132596da9a3b30d83af2c0908f8fc862b66ef3dd"
Sep 17 14:31:14.572549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:14.572521    2565 generic.go:356] "Generic (PLEG): container finished" podID="ff41f195-e3b8-4886-8e6d-038f2e6c596c" containerID="5da15f8a0273dc31ab5deead45a3c7b321b3f264289eaf255ff8295eef1e4fc0" exitCode=0
Sep 17 14:31:14.572673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:14.572599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl" event={"ID":"ff41f195-e3b8-4886-8e6d-038f2e6c596c","Type":"ContainerDied","Data":"5da15f8a0273dc31ab5deead45a3c7b321b3f264289eaf255ff8295eef1e4fc0"}
Sep 17 14:31:14.622058 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:14.622024    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"]
Sep 17 14:31:14.626653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:14.626627    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/e2e-conflict-a-4102a62c-kserve-7fdc475c4-w5kgs"]
Sep 17 14:31:15.153107 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:15.153071    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:15.520740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:15.520655    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc5096ed-0d69-4616-8f9f-3663dfbf940d" path="/var/lib/kubelet/pods/bc5096ed-0d69-4616-8f9f-3663dfbf940d/volumes"
Sep 17 14:31:15.719695 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:15.719672    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"
Sep 17 14:31:15.742574 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:15.742551    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff41f195-e3b8-4886-8e6d-038f2e6c596c-maas-api-service-ca\") pod \"ff41f195-e3b8-4886-8e6d-038f2e6c596c\" (UID: \"ff41f195-e3b8-4886-8e6d-038f2e6c596c\") "
Sep 17 14:31:15.742881 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:15.742857    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff41f195-e3b8-4886-8e6d-038f2e6c596c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff41f195-e3b8-4886-8e6d-038f2e6c596c" (UID: "ff41f195-e3b8-4886-8e6d-038f2e6c596c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:15.742981 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:15.742964    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff41f195-e3b8-4886-8e6d-038f2e6c596c-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:15.910547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:15.910510    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:16.586221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:16.586183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl" event={"ID":"ff41f195-e3b8-4886-8e6d-038f2e6c596c","Type":"ContainerDied","Data":"d3e9da63e7bf0c1319d6b8f54306a52c779c0f836bdd468d056284e632590601"}
Sep 17 14:31:16.586221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:16.586217    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"
Sep 17 14:31:16.586666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:16.586242    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3e9da63e7bf0c1319d6b8f54306a52c779c0f836bdd468d056284e632590601"
Sep 17 14:31:16.759652 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:16.759624    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:17.512690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:17.512658    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:18.491053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:18.491019    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:19.273227 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:19.273196    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:19.994114 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:19.994082    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:28.392901 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.392865    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"]
Sep 17 14:31:28.395782 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.395751    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:28.396700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.396675    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2d17aff-29827590-spvr7"]
Sep 17 14:31:28.427711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.427678    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"]
Sep 17 14:31:28.427966 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.427936    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh" podUID="69cc0e3b-a67a-42d2-bb00-42b9175513d4" containerName="payload-processing" containerID="cri-o://f7a0d3437e7b7fa6ebb098566eb2037b8a7c677313bdc827e124b1e7893c0f20" gracePeriod=30
Sep 17 14:31:28.471431 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.471400    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:28.574608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.574578    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:28.620451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.620420    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"]
Sep 17 14:31:28.625972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.625915    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d2d17aff-l7jsl"]
Sep 17 14:31:28.662198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.662132    2565 generic.go:356] "Generic (PLEG): container finished" podID="69cc0e3b-a67a-42d2-bb00-42b9175513d4" containerID="f7a0d3437e7b7fa6ebb098566eb2037b8a7c677313bdc827e124b1e7893c0f20" exitCode=0
Sep 17 14:31:28.662354 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.662200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh" event={"ID":"69cc0e3b-a67a-42d2-bb00-42b9175513d4","Type":"ContainerDied","Data":"f7a0d3437e7b7fa6ebb098566eb2037b8a7c677313bdc827e124b1e7893c0f20"}
Sep 17 14:31:28.673380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.673357    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:28.686097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.686080    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:31:28.781279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.781221    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5sqk5\" (UniqueName: \"kubernetes.io/projected/69cc0e3b-a67a-42d2-bb00-42b9175513d4-kube-api-access-5sqk5\") pod \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\" (UID: \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\") "
Sep 17 14:31:28.781521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.781496    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cc0e3b-a67a-42d2-bb00-42b9175513d4-plugins-config-volume\") pod \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\" (UID: \"69cc0e3b-a67a-42d2-bb00-42b9175513d4\") "
Sep 17 14:31:28.781864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.781838    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cc0e3b-a67a-42d2-bb00-42b9175513d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69cc0e3b-a67a-42d2-bb00-42b9175513d4" (UID: "69cc0e3b-a67a-42d2-bb00-42b9175513d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:31:28.782049 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.782032    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69cc0e3b-a67a-42d2-bb00-42b9175513d4-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:28.783932 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.783907    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cc0e3b-a67a-42d2-bb00-42b9175513d4-kube-api-access-5sqk5" (OuterVolumeSpecName: "kube-api-access-5sqk5") pod "69cc0e3b-a67a-42d2-bb00-42b9175513d4" (UID: "69cc0e3b-a67a-42d2-bb00-42b9175513d4"). InnerVolumeSpecName "kube-api-access-5sqk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:28.883100 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:28.883065    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sqk5\" (UniqueName: \"kubernetes.io/projected/69cc0e3b-a67a-42d2-bb00-42b9175513d4-kube-api-access-5sqk5\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:29.127844 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:29.127805    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:29.520767 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:29.520685    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="783fe4d4-3c4a-41fd-ad3d-2b01090dca0f" path="/var/lib/kubelet/pods/783fe4d4-3c4a-41fd-ad3d-2b01090dca0f/volumes"
Sep 17 14:31:29.521123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:29.521011    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff41f195-e3b8-4886-8e6d-038f2e6c596c" path="/var/lib/kubelet/pods/ff41f195-e3b8-4886-8e6d-038f2e6c596c/volumes"
Sep 17 14:31:29.669789 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:29.669758    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"
Sep 17 14:31:29.669964 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:29.669766    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh" event={"ID":"69cc0e3b-a67a-42d2-bb00-42b9175513d4","Type":"ContainerDied","Data":"dda07490a846da2b30e02ab3125df636dbef30dc1f33daf812ff2c7856c85bbe"}
Sep 17 14:31:29.669964 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:29.669889    2565 scope.go:117] "RemoveContainer" containerID="f7a0d3437e7b7fa6ebb098566eb2037b8a7c677313bdc827e124b1e7893c0f20"
Sep 17 14:31:29.694763 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:29.694737    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"]
Sep 17 14:31:29.699172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:29.699148    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d2d17aff-676cd69d4f-sbcrh"]
Sep 17 14:31:30.049175 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:30.049138    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:31.046989 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:31.046952    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:31.520604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:31.520555    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69cc0e3b-a67a-42d2-bb00-42b9175513d4" path="/var/lib/kubelet/pods/69cc0e3b-a67a-42d2-bb00-42b9175513d4/volumes"
Sep 17 14:31:31.845024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:31.844995    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:31.974985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:31.974951    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:32.742362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:32.742329    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:33.518471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:33.518437    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:34.256236 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:34.256200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:36.823384 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:36.823355    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"]
Sep 17 14:31:36.915814 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:36.915778    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:37.522396 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522361    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"]
Sep 17 14:31:37.522787 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522774    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc5096ed-0d69-4616-8f9f-3663dfbf940d" containerName="main"
Sep 17 14:31:37.522833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522789    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc5096ed-0d69-4616-8f9f-3663dfbf940d" containerName="main"
Sep 17 14:31:37.522833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522800    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cc0e3b-a67a-42d2-bb00-42b9175513d4" containerName="payload-processing"
Sep 17 14:31:37.522833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522805    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cc0e3b-a67a-42d2-bb00-42b9175513d4" containerName="payload-processing"
Sep 17 14:31:37.522833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522816    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff41f195-e3b8-4886-8e6d-038f2e6c596c" containerName="revoke-keys"
Sep 17 14:31:37.522833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522822    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff41f195-e3b8-4886-8e6d-038f2e6c596c" containerName="revoke-keys"
Sep 17 14:31:37.522994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522889    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff41f195-e3b8-4886-8e6d-038f2e6c596c" containerName="revoke-keys"
Sep 17 14:31:37.522994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522898    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="69cc0e3b-a67a-42d2-bb00-42b9175513d4" containerName="payload-processing"
Sep 17 14:31:37.522994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.522904    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc5096ed-0d69-4616-8f9f-3663dfbf940d" containerName="main"
Sep 17 14:31:37.527228 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.527206    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.530796 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.530767    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"openshift-service-ca.crt\""
Sep 17 14:31:37.531536 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.531513    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"default-dockercfg-8klfk\""
Sep 17 14:31:37.531536 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.531534    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"auth-test-model-3da4be-kserve-self-signed-certs\""
Sep 17 14:31:37.531820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.531513    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"kube-root-ca.crt\""
Sep 17 14:31:37.536573 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.536553    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"]
Sep 17 14:31:37.683376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.683340    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tls-certs\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.683588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.683423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tmp-dir\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.683588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.683529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-model-cache\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.683714 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.683594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-home\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.683714 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.683673    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vspfj\" (UniqueName: \"kubernetes.io/projected/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-kube-api-access-vspfj\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.683714 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.683701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-dshm\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.785769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.784825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tmp-dir\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.785769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.784927    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-model-cache\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.785769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.784983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-home\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.785769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.785074    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vspfj\" (UniqueName: \"kubernetes.io/projected/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-kube-api-access-vspfj\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.785769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.785101    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-dshm\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.785769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.785136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tls-certs\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.786702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.786671    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tmp-dir\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.787116 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.786818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-model-cache\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.787286 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.787059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-home\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.789352 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.789328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-dshm\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.790023 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.790001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tls-certs\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.796614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.796565    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vspfj\" (UniqueName: \"kubernetes.io/projected/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-kube-api-access-vspfj\") pod \"auth-test-model-3da4be-kserve-5f564b7747-qgphx\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:37.838459 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:37.838425    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:38.119222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:38.119192    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"]
Sep 17 14:31:38.120895 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:31:38.120862    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3ca3153e_7e2e_4c8b_9c10_058f04c2d359.slice/crio-5954d904b200f00246397c0cc2c5bb6246034425f0ca7892974ac02bfceccc44 WatchSource:0}: Error finding container 5954d904b200f00246397c0cc2c5bb6246034425f0ca7892974ac02bfceccc44: Status 404 returned error can't find the container with id 5954d904b200f00246397c0cc2c5bb6246034425f0ca7892974ac02bfceccc44
Sep 17 14:31:38.193569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:38.193545    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:38.737028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:38.736974    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx" event={"ID":"3ca3153e-7e2e-4c8b-9c10-058f04c2d359","Type":"ContainerStarted","Data":"107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80"}
Sep 17 14:31:38.737370 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:38.737335    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx" event={"ID":"3ca3153e-7e2e-4c8b-9c10-058f04c2d359","Type":"ContainerStarted","Data":"5954d904b200f00246397c0cc2c5bb6246034425f0ca7892974ac02bfceccc44"}
Sep 17 14:31:38.764848 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:38.764788    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx" podStartSLOduration=1.764767389 podStartE2EDuration="1.764767389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:31:38.758478649 +0000 UTC m=+1485.691983842" watchObservedRunningTime="2026-09-17 14:31:38.764767389 +0000 UTC m=+1485.698272582"
Sep 17 14:31:46.525450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:46.525419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:47.357538 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:47.357503    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:47.437221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:47.437183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:31:47.838934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:47.838889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:47.838934 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:47.838937    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:47.852722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:47.852681    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:48.811170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:48.811130    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:31:51.842098 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:51.842058    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2" podUID="a11a370f-c8f1-4819-9311-838f877f13c2" containerName="main" containerID="cri-o://179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f" gracePeriod=45
Sep 17 14:31:52.095558 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.095493    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:31:52.140349 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140314    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6krs\" (UniqueName: \"kubernetes.io/projected/a11a370f-c8f1-4819-9311-838f877f13c2-kube-api-access-w6krs\") pod \"a11a370f-c8f1-4819-9311-838f877f13c2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") "
Sep 17 14:31:52.140517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140366    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-home\") pod \"a11a370f-c8f1-4819-9311-838f877f13c2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") "
Sep 17 14:31:52.140517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140389    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-dshm\") pod \"a11a370f-c8f1-4819-9311-838f877f13c2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") "
Sep 17 14:31:52.140517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140417    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a370f-c8f1-4819-9311-838f877f13c2-tls-certs\") pod \"a11a370f-c8f1-4819-9311-838f877f13c2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") "
Sep 17 14:31:52.140517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140476    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-tmp-dir\") pod \"a11a370f-c8f1-4819-9311-838f877f13c2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") "
Sep 17 14:31:52.140781 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140593    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-model-cache\") pod \"a11a370f-c8f1-4819-9311-838f877f13c2\" (UID: \"a11a370f-c8f1-4819-9311-838f877f13c2\") "
Sep 17 14:31:52.140987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140855    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a11a370f-c8f1-4819-9311-838f877f13c2" (UID: "a11a370f-c8f1-4819-9311-838f877f13c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:31:52.140987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140906    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-home" (OuterVolumeSpecName: "home") pod "a11a370f-c8f1-4819-9311-838f877f13c2" (UID: "a11a370f-c8f1-4819-9311-838f877f13c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:31:52.140987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.140932    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-model-cache" (OuterVolumeSpecName: "model-cache") pod "a11a370f-c8f1-4819-9311-838f877f13c2" (UID: "a11a370f-c8f1-4819-9311-838f877f13c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:31:52.141185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.141095    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:52.141185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.141115    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:52.141185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.141127    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:52.143207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.143159    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-dshm" (OuterVolumeSpecName: "dshm") pod "a11a370f-c8f1-4819-9311-838f877f13c2" (UID: "a11a370f-c8f1-4819-9311-838f877f13c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:31:52.143207 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.143180    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11a370f-c8f1-4819-9311-838f877f13c2-kube-api-access-w6krs" (OuterVolumeSpecName: "kube-api-access-w6krs") pod "a11a370f-c8f1-4819-9311-838f877f13c2" (UID: "a11a370f-c8f1-4819-9311-838f877f13c2"). InnerVolumeSpecName "kube-api-access-w6krs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:31:52.143416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.143246    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a11a370f-c8f1-4819-9311-838f877f13c2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a11a370f-c8f1-4819-9311-838f877f13c2" (UID: "a11a370f-c8f1-4819-9311-838f877f13c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:31:52.241890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.241862    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6krs\" (UniqueName: \"kubernetes.io/projected/a11a370f-c8f1-4819-9311-838f877f13c2-kube-api-access-w6krs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:52.241890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.241888    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a11a370f-c8f1-4819-9311-838f877f13c2-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:52.241890 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.241898    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a11a370f-c8f1-4819-9311-838f877f13c2-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:31:52.826300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.826265    2565 generic.go:356] "Generic (PLEG): container finished" podID="a11a370f-c8f1-4819-9311-838f877f13c2" containerID="179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f" exitCode=0
Sep 17 14:31:52.826300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.826299    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"
Sep 17 14:31:52.826524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.826347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2" event={"ID":"a11a370f-c8f1-4819-9311-838f877f13c2","Type":"ContainerDied","Data":"179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f"}
Sep 17 14:31:52.826524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.826389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2" event={"ID":"a11a370f-c8f1-4819-9311-838f877f13c2","Type":"ContainerDied","Data":"625d86f88c10d535397aa6dd792956ad586d6d6775a7edb912f9a4f09837797b"}
Sep 17 14:31:52.826524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.826406    2565 scope.go:117] "RemoveContainer" containerID="179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f"
Sep 17 14:31:52.836599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.836585    2565 scope.go:117] "RemoveContainer" containerID="179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f"
Sep 17 14:31:52.836876 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:31:52.836856    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f\": container with ID starting with 179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f not found: ID does not exist" containerID="179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f"
Sep 17 14:31:52.836953 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.836889    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f"} err="failed to get container status \"179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f\": rpc error: code = NotFound desc = could not find container \"179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f\": container with ID starting with 179058b88555e3bd75f058c8bfd6cfd625fa65850e9324d137639c445650858f not found: ID does not exist"
Sep 17 14:31:52.850539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.850517    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"]
Sep 17 14:31:52.854038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:52.854016    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d2d17aff/e2e-lifecycle-model-d2d17aff-kserve-7449bb9679-sz4r2"]
Sep 17 14:31:53.520648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:53.520617    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11a370f-c8f1-4819-9311-838f877f13c2" path="/var/lib/kubelet/pods/a11a370f-c8f1-4819-9311-838f877f13c2/volumes"
Sep 17 14:31:53.579185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:53.579156    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:31:53.584941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:53.584917    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:31:53.585170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:53.585151    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:31:53.590860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:53.590845    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:31:53.807844 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:31:53.807761    2565 scope.go:117] "RemoveContainer" containerID="e76af364f6f580a1412b53713c47e8a3ae405f500a170084e239eda899d85714"
Sep 17 14:32:12.896843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:12.896794    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:13.685854 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:13.685820    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:14.603467 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:14.603432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:20.423698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.423596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:20.929864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.929832    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"]
Sep 17 14:32:20.930328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.930314    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11a370f-c8f1-4819-9311-838f877f13c2" containerName="main"
Sep 17 14:32:20.930383 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.930330    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11a370f-c8f1-4819-9311-838f877f13c2" containerName="main"
Sep 17 14:32:20.930420 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.930412    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11a370f-c8f1-4819-9311-838f877f13c2" containerName="main"
Sep 17 14:32:20.933751 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.933729    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:20.936360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.936333    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"kube-root-ca.crt\""
Sep 17 14:32:20.936931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.936913    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"auth-test-model-b48b7c-kserve-self-signed-certs\""
Sep 17 14:32:20.937004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.936899    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"openshift-service-ca.crt\""
Sep 17 14:32:20.937053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.937027    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"default-dockercfg-7ltgp\""
Sep 17 14:32:20.943986 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.943959    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"]
Sep 17 14:32:20.958382 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.958353    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-tmp-dir\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:20.958512 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.958396    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-model-cache\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:20.958512 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.958457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38aefff7-cf34-474b-9b39-c6a10d1390d3-tls-certs\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:20.958512 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.958506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-dshm\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:20.958690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.958651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-home\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:20.958752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.958700    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dtn\" (UniqueName: \"kubernetes.io/projected/38aefff7-cf34-474b-9b39-c6a10d1390d3-kube-api-access-h5dtn\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:20.964815 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:20.964772    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:21.060411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.060368    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-home\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.060582 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.060529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dtn\" (UniqueName: \"kubernetes.io/projected/38aefff7-cf34-474b-9b39-c6a10d1390d3-kube-api-access-h5dtn\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.060672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.060657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-tmp-dir\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.060711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.060684    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-model-cache\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.060711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.060701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38aefff7-cf34-474b-9b39-c6a10d1390d3-tls-certs\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.060824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.060730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-dshm\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.060824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.060784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-home\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.061001 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.060976    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-tmp-dir\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.061204 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.061185    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-model-cache\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.063148 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.063114    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-dshm\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.063332 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.063313    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38aefff7-cf34-474b-9b39-c6a10d1390d3-tls-certs\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.070423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.070397    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dtn\" (UniqueName: \"kubernetes.io/projected/38aefff7-cf34-474b-9b39-c6a10d1390d3-kube-api-access-h5dtn\") pod \"auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.246895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.246805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:21.532895 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:21.532868    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"]
Sep 17 14:32:21.535161 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:32:21.535130    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38aefff7_cf34_474b_9b39_c6a10d1390d3.slice/crio-1eeb0474a9af699e378252505268f090dc06fb2f3e4ce00cf1a1b482f8d5240e WatchSource:0}: Error finding container 1eeb0474a9af699e378252505268f090dc06fb2f3e4ce00cf1a1b482f8d5240e: Status 404 returned error can't find the container with id 1eeb0474a9af699e378252505268f090dc06fb2f3e4ce00cf1a1b482f8d5240e
Sep 17 14:32:22.013329 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:22.013285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt" event={"ID":"38aefff7-cf34-474b-9b39-c6a10d1390d3","Type":"ContainerStarted","Data":"506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03"}
Sep 17 14:32:22.013535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:22.013342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt" event={"ID":"38aefff7-cf34-474b-9b39-c6a10d1390d3","Type":"ContainerStarted","Data":"1eeb0474a9af699e378252505268f090dc06fb2f3e4ce00cf1a1b482f8d5240e"}
Sep 17 14:32:22.034113 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:22.034061    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt" podStartSLOduration=2.034045168 podStartE2EDuration="2.034045168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:32:22.033052827 +0000 UTC m=+1528.966558013" watchObservedRunningTime="2026-09-17 14:32:22.034045168 +0000 UTC m=+1528.967550359"
Sep 17 14:32:23.410409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:23.410347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:23.727560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:23.727454    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:25.332330 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:25.332248    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:31.247156 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:31.247114    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:31.247156 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:31.247164    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:31.264352 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:31.264327    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:31.468199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:31.468165    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:32.092596 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:32.092564    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:32:54.510669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.510622    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"]
Sep 17 14:32:54.514620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.514592    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.517333 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.517306    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b50f2\"/\"distinct-2-w4-8b50f2-kserve-self-signed-certs\""
Sep 17 14:32:54.524174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.524146    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"]
Sep 17 14:32:54.539927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.539888    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:32:54.627801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.627758    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-dshm\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.628004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.627847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw7gx\" (UniqueName: \"kubernetes.io/projected/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-kube-api-access-gw7gx\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.628004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.627893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-home\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.628004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.627976    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tls-certs\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.628164 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.628074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-model-cache\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.628164 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.628100    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tmp-dir\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.729468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.729426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-home\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.729468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.729473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tls-certs\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.729715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.729544    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-model-cache\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.729715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.729568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tmp-dir\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.729715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.729623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-dshm\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.729715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.729668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw7gx\" (UniqueName: \"kubernetes.io/projected/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-kube-api-access-gw7gx\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.729965 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.729939    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-home\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.730038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.729948    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-model-cache\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.730085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.730037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tmp-dir\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.731863 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.731837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-dshm\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.732071 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.732055    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tls-certs\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.739189 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.739155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw7gx\" (UniqueName: \"kubernetes.io/projected/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-kube-api-access-gw7gx\") pod \"distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") " pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:54.825163 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:54.825136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:32:55.101892 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:55.101865    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"]
Sep 17 14:32:55.103372 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:32:55.103345    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9f89105_a7b9_4f2b_b445_0dc45f5a06de.slice/crio-42864058c2805b9aa79fd397d3efd975f8f0c2be32a1cf0008990f8d396b910d WatchSource:0}: Error finding container 42864058c2805b9aa79fd397d3efd975f8f0c2be32a1cf0008990f8d396b910d: Status 404 returned error can't find the container with id 42864058c2805b9aa79fd397d3efd975f8f0c2be32a1cf0008990f8d396b910d
Sep 17 14:32:55.224395 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:55.224354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr" event={"ID":"e9f89105-a7b9-4f2b-b445-0dc45f5a06de","Type":"ContainerStarted","Data":"dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7"}
Sep 17 14:32:55.224591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:55.224404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr" event={"ID":"e9f89105-a7b9-4f2b-b445-0dc45f5a06de","Type":"ContainerStarted","Data":"42864058c2805b9aa79fd397d3efd975f8f0c2be32a1cf0008990f8d396b910d"}
Sep 17 14:32:55.248123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:32:55.248062    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr" podStartSLOduration=1.248045267 podStartE2EDuration="1.248045267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:32:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:32:55.247508415 +0000 UTC m=+1562.181013610" watchObservedRunningTime="2026-09-17 14:32:55.248045267 +0000 UTC m=+1562.181550461"
Sep 17 14:33:02.067982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:02.067942    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"]
Sep 17 14:33:02.068493 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:02.068269    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy" containerID="cri-o://b1803658ad3f58c0e99f758c3e2e41d4ff50add930f2bdd10c2cf760b66bcc6e" gracePeriod=30
Sep 17 14:33:03.167823 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:03.167789    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:03.308465 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:03.308432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:03.453823 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:03.453723    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:03.551580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:03.551545    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:04.230546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:04.230512    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:04.270313 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:04.270247    2565 patch_prober.go:28] interesting pod/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:33:04.270496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:04.270333    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:33:04.338592 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:04.338558    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:04.825499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:04.825463    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:33:04.825679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:04.825520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:33:04.838928 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:04.838899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:33:05.300700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:05.300618    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:33:05.737787 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:05.737750    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:07.305984 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.305943    2565 generic.go:356] "Generic (PLEG): container finished" podID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerID="b1803658ad3f58c0e99f758c3e2e41d4ff50add930f2bdd10c2cf760b66bcc6e" exitCode=0
Sep 17 14:33:07.306475 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.306046    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" event={"ID":"3fa2b461-0299-4edf-a657-fb8e646c6398","Type":"ContainerDied","Data":"b1803658ad3f58c0e99f758c3e2e41d4ff50add930f2bdd10c2cf760b66bcc6e"}
Sep 17 14:33:07.306475 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.306129    2565 scope.go:117] "RemoveContainer" containerID="d95768977be315cee6fe711bdf2ac96266e341e880587a3478cdb1d506db9c2b"
Sep 17 14:33:07.361674 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.361640    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:33:07.484115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484078    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-token\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484327 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484170    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2b461-0299-4edf-a657-fb8e646c6398-istiod-ca-cert\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484327 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484209    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-socket\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484327 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484293    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-credential-socket\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484327 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484311    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-podinfo\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-envoy\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484446    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-certs\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484482    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-data\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484555 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8bnl\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-kube-api-access-v8bnl\") pod \"3fa2b461-0299-4edf-a657-fb8e646c6398\" (UID: \"3fa2b461-0299-4edf-a657-fb8e646c6398\") "
Sep 17 14:33:07.484780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:07.484780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484714    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:07.484780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484649    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fa2b461-0299-4edf-a657-fb8e646c6398-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:33:07.484952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484827    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-data" (OuterVolumeSpecName: "istio-data") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:07.485008 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.484952    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:07.485118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.485094    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:07.485118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.485120    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:07.485351 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.485135    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fa2b461-0299-4edf-a657-fb8e646c6398-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:07.485351 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.485149    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:07.485351 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.485162    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:07.486611 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.486583    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:33:07.486740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.486693    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:33:07.486828 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.486805    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-token" (OuterVolumeSpecName: "istio-token") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:07.486933 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.486914    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-kube-api-access-v8bnl" (OuterVolumeSpecName: "kube-api-access-v8bnl") pod "3fa2b461-0299-4edf-a657-fb8e646c6398" (UID: "3fa2b461-0299-4edf-a657-fb8e646c6398"). InnerVolumeSpecName "kube-api-access-v8bnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:33:07.586672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.586578    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8bnl\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-kube-api-access-v8bnl\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:07.586672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.586612    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:07.586672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.586625    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:07.586672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:07.586634    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fa2b461-0299-4edf-a657-fb8e646c6398-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:33:08.313760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:08.313728    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"
Sep 17 14:33:08.314272 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:08.313771    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq" event={"ID":"3fa2b461-0299-4edf-a657-fb8e646c6398","Type":"ContainerDied","Data":"e6c0af248e59bd44cffb56c40305793fc8d235de9703e8914d25c0d78d3f980b"}
Sep 17 14:33:08.314272 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:08.313832    2565 scope.go:117] "RemoveContainer" containerID="b1803658ad3f58c0e99f758c3e2e41d4ff50add930f2bdd10c2cf760b66bcc6e"
Sep 17 14:33:08.333161 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:08.333135    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"]
Sep 17 14:33:08.336984 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:08.336955    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d2d17aff-openshift-default-74687dc448-c6drq"]
Sep 17 14:33:09.084884 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:09.084852    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:09.520213 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:09.520111    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" path="/var/lib/kubelet/pods/3fa2b461-0299-4edf-a657-fb8e646c6398/volumes"
Sep 17 14:33:11.565299 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:11.565239    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:11.650151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:11.650121    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:12.247589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:12.247556    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:18.125796 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:18.125762    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:18.237939 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:18.237909    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:18.340887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:18.340851    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:28.911596 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:28.911558    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:29.074539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:29.074495    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:41.153700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:41.153667    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:49.963186 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:49.963109    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:53.852732 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:53.852695    2565 scope.go:117] "RemoveContainer" containerID="f0b187975c2aac91158f425a9a401022347fd71134579906ae1313358bf30227"
Sep 17 14:33:53.862593 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:53.862569    2565 scope.go:117] "RemoveContainer" containerID="5872b646059ed798135ea222e02216f5ab7e124c582434189ec826bbb2b8fcf2"
Sep 17 14:33:55.816298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:55.816233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:55.879617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:55.879582    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:56.151012 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:56.150970    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:56.450973 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:56.450877    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:56.591340 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:56.591300    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:58.511629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:58.511582    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:59.184198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:59.184165    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:59.274646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:59.274611    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:33:59.377980 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:33:59.377947    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:00.140773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:00.140740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:00.825203 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:00.825167    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:01.612081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:01.612050    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:02.272777 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:02.272744    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:02.920098 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:02.920056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:03.754461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:03.754429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:03.854485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:03.854458    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:04.049712 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.049624    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:04.223543 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.223510    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:04.329769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.329741    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:04.362677 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.362648    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"]
Sep 17 14:34:04.363099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.363086    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy"
Sep 17 14:34:04.363152 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.363100    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy"
Sep 17 14:34:04.363152 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.363115    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy"
Sep 17 14:34:04.363152 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.363120    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy"
Sep 17 14:34:04.363304 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.363199    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy"
Sep 17 14:34:04.366044 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.366017    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:04.371008 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.370989    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"]
Sep 17 14:34:04.371496 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.371481    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fa2b461-0299-4edf-a657-fb8e646c6398" containerName="istio-proxy"
Sep 17 14:34:04.374804 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.374785    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:04.375991 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.375975    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-29d3c7-dockercfg-t24sm\""
Sep 17 14:34:04.377286 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.377267    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-29d3c7\""
Sep 17 14:34:04.392598 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.392573    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"]
Sep 17 14:34:04.399566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.399543    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"]
Sep 17 14:34:04.503306 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.503239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8643c32-3647-4c36-90f6-c2896ec66f08-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs\" (UID: \"e8643c32-3647-4c36-90f6-c2896ec66f08\") " pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:04.503518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.503330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j97t\" (UniqueName: \"kubernetes.io/projected/e8643c32-3647-4c36-90f6-c2896ec66f08-kube-api-access-8j97t\") pod \"payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs\" (UID: \"e8643c32-3647-4c36-90f6-c2896ec66f08\") " pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:04.503518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.503370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-997nf\" (UniqueName: \"kubernetes.io/projected/7459ff24-4a73-484b-a562-720925fed82b-kube-api-access-997nf\") pod \"payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f\" (UID: \"7459ff24-4a73-484b-a562-720925fed82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:04.503518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.503425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ff24-4a73-484b-a562-720925fed82b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f\" (UID: \"7459ff24-4a73-484b-a562-720925fed82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:04.605089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.604967    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8643c32-3647-4c36-90f6-c2896ec66f08-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs\" (UID: \"e8643c32-3647-4c36-90f6-c2896ec66f08\") " pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:04.605089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.605035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j97t\" (UniqueName: \"kubernetes.io/projected/e8643c32-3647-4c36-90f6-c2896ec66f08-kube-api-access-8j97t\") pod \"payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs\" (UID: \"e8643c32-3647-4c36-90f6-c2896ec66f08\") " pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:04.605089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.605069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-997nf\" (UniqueName: \"kubernetes.io/projected/7459ff24-4a73-484b-a562-720925fed82b-kube-api-access-997nf\") pod \"payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f\" (UID: \"7459ff24-4a73-484b-a562-720925fed82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:04.605474 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.605110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ff24-4a73-484b-a562-720925fed82b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f\" (UID: \"7459ff24-4a73-484b-a562-720925fed82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:04.605924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.605898    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ff24-4a73-484b-a562-720925fed82b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f\" (UID: \"7459ff24-4a73-484b-a562-720925fed82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:04.606028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.605906    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8643c32-3647-4c36-90f6-c2896ec66f08-plugins-config-volume\") pod \"payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs\" (UID: \"e8643c32-3647-4c36-90f6-c2896ec66f08\") " pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:04.617266 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.617222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j97t\" (UniqueName: \"kubernetes.io/projected/e8643c32-3647-4c36-90f6-c2896ec66f08-kube-api-access-8j97t\") pod \"payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs\" (UID: \"e8643c32-3647-4c36-90f6-c2896ec66f08\") " pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:04.622610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.622590    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-997nf\" (UniqueName: \"kubernetes.io/projected/7459ff24-4a73-484b-a562-720925fed82b-kube-api-access-997nf\") pod \"payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f\" (UID: \"7459ff24-4a73-484b-a562-720925fed82b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:04.677138 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.677108    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:04.685597 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.685575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:04.961791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.961759    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"]
Sep 17 14:34:04.962665 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:34:04.962637    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7459ff24_4a73_484b_a562_720925fed82b.slice/crio-5fb0ef1fb265ed54b7ed708d6bc19edb5add6916b59209c74bb406c53d16db3a WatchSource:0}: Error finding container 5fb0ef1fb265ed54b7ed708d6bc19edb5add6916b59209c74bb406c53d16db3a: Status 404 returned error can't find the container with id 5fb0ef1fb265ed54b7ed708d6bc19edb5add6916b59209c74bb406c53d16db3a
Sep 17 14:34:04.971472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:04.971447    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"]
Sep 17 14:34:04.972540 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:34:04.972515    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8643c32_3647_4c36_90f6_c2896ec66f08.slice/crio-ce5ac42eb0e805514b6a86a7d238c5d58f9dcac5d158dfd714e908ed01aaece7 WatchSource:0}: Error finding container ce5ac42eb0e805514b6a86a7d238c5d58f9dcac5d158dfd714e908ed01aaece7: Status 404 returned error can't find the container with id ce5ac42eb0e805514b6a86a7d238c5d58f9dcac5d158dfd714e908ed01aaece7
Sep 17 14:34:05.333562 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.333516    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:05.676874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.676769    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs" event={"ID":"e8643c32-3647-4c36-90f6-c2896ec66f08","Type":"ContainerStarted","Data":"2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb"}
Sep 17 14:34:05.676874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.676839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs" event={"ID":"e8643c32-3647-4c36-90f6-c2896ec66f08","Type":"ContainerStarted","Data":"ce5ac42eb0e805514b6a86a7d238c5d58f9dcac5d158dfd714e908ed01aaece7"}
Sep 17 14:34:05.676874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.676871    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:05.678697 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.678671    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f" event={"ID":"7459ff24-4a73-484b-a562-720925fed82b","Type":"ContainerStarted","Data":"c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e"}
Sep 17 14:34:05.678827 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.678718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f" event={"ID":"7459ff24-4a73-484b-a562-720925fed82b","Type":"ContainerStarted","Data":"5fb0ef1fb265ed54b7ed708d6bc19edb5add6916b59209c74bb406c53d16db3a"}
Sep 17 14:34:05.678827 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.678748    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:05.696083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.696034    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs" podStartSLOduration=1.696016113 podStartE2EDuration="1.696016113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:34:05.694434156 +0000 UTC m=+1632.627939349" watchObservedRunningTime="2026-09-17 14:34:05.696016113 +0000 UTC m=+1632.629521309"
Sep 17 14:34:05.715056 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:05.715011    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f" podStartSLOduration=1.714993125 podStartE2EDuration="1.714993125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:34:05.714645354 +0000 UTC m=+1632.648150546" watchObservedRunningTime="2026-09-17 14:34:05.714993125 +0000 UTC m=+1632.648498317"
Sep 17 14:34:06.079102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:06.079063    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:16.695235 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:16.695193    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:34:16.695823 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:16.695398    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:34:18.866927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:18.866886    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:24.089186 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.089153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:24.200055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.200023    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"]
Sep 17 14:34:24.203954 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.203934    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"
Sep 17 14:34:24.206702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.206678    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:34:24.207198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.207183    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:34:24.216450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.216423    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"]
Sep 17 14:34:24.341237 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.341153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06fe340-d231-4fcd-ae67-47e7c4ac9781-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5\" (UID: \"d06fe340-d231-4fcd-ae67-47e7c4ac9781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"
Sep 17 14:34:24.442383 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.442351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06fe340-d231-4fcd-ae67-47e7c4ac9781-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5\" (UID: \"d06fe340-d231-4fcd-ae67-47e7c4ac9781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"
Sep 17 14:34:24.442975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.442955    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06fe340-d231-4fcd-ae67-47e7c4ac9781-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5\" (UID: \"d06fe340-d231-4fcd-ae67-47e7c4ac9781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"
Sep 17 14:34:24.515983 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.515960    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"
Sep 17 14:34:24.564629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.564597    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:24.792442 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.792412    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"]
Sep 17 14:34:24.793929 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:34:24.793903    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06fe340_d231_4fcd_ae67_47e7c4ac9781.slice/crio-c04ae4458762c5dfacd08e9ecd7a7b98bb387d6e21504c0b68e728687049f531 WatchSource:0}: Error finding container c04ae4458762c5dfacd08e9ecd7a7b98bb387d6e21504c0b68e728687049f531: Status 404 returned error can't find the container with id c04ae4458762c5dfacd08e9ecd7a7b98bb387d6e21504c0b68e728687049f531
Sep 17 14:34:24.809215 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:24.809188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5" event={"ID":"d06fe340-d231-4fcd-ae67-47e7c4ac9781","Type":"ContainerStarted","Data":"c04ae4458762c5dfacd08e9ecd7a7b98bb387d6e21504c0b68e728687049f531"}
Sep 17 14:34:25.492968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:25.492935    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:25.816644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:25.816549    2565 generic.go:356] "Generic (PLEG): container finished" podID="d06fe340-d231-4fcd-ae67-47e7c4ac9781" containerID="15ea0f4b2938c93e69e32be1855ecdb088d31197c2c3790e9152ae63890f9644" exitCode=0
Sep 17 14:34:25.816811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:25.816638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5" event={"ID":"d06fe340-d231-4fcd-ae67-47e7c4ac9781","Type":"ContainerDied","Data":"15ea0f4b2938c93e69e32be1855ecdb088d31197c2c3790e9152ae63890f9644"}
Sep 17 14:34:26.279745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:26.279710    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:26.961061 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:26.961034    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"
Sep 17 14:34:27.072849 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:27.072823    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06fe340-d231-4fcd-ae67-47e7c4ac9781-maas-api-service-ca\") pod \"d06fe340-d231-4fcd-ae67-47e7c4ac9781\" (UID: \"d06fe340-d231-4fcd-ae67-47e7c4ac9781\") "
Sep 17 14:34:27.073303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:27.073247    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06fe340-d231-4fcd-ae67-47e7c4ac9781-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d06fe340-d231-4fcd-ae67-47e7c4ac9781" (UID: "d06fe340-d231-4fcd-ae67-47e7c4ac9781"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:34:27.174702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:27.174668    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d06fe340-d231-4fcd-ae67-47e7c4ac9781-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:27.436773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:27.436681    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:27.828779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:27.828749    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"
Sep 17 14:34:27.828980 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:27.828791    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5" event={"ID":"d06fe340-d231-4fcd-ae67-47e7c4ac9781","Type":"ContainerDied","Data":"c04ae4458762c5dfacd08e9ecd7a7b98bb387d6e21504c0b68e728687049f531"}
Sep 17 14:34:27.828980 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:27.828834    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c04ae4458762c5dfacd08e9ecd7a7b98bb387d6e21504c0b68e728687049f531"
Sep 17 14:34:28.166845 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:28.166740    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:29.038625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:29.038587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:29.154406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:29.154372    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"]
Sep 17 14:34:29.253658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:29.253626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:29.807359 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:29.807295    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:30.629268 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:30.629170    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:31.423374 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:31.423343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:32.133118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:32.133088    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:32.846082 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:32.846047    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:33.739381 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:33.739350    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:34.332133 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:34.332096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:34.443284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:34.443236    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:34.760860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:34.760765    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:35.025336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.025226    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:35.082815 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.082782    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"]
Sep 17 14:34:35.083348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.083329    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06fe340-d231-4fcd-ae67-47e7c4ac9781" containerName="revoke-keys"
Sep 17 14:34:35.083348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.083348    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06fe340-d231-4fcd-ae67-47e7c4ac9781" containerName="revoke-keys"
Sep 17 14:34:35.083495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.083436    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06fe340-d231-4fcd-ae67-47e7c4ac9781" containerName="revoke-keys"
Sep 17 14:34:35.086314 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.086288    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.091111 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.091090    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0829c1-openshift-default-dockercfg-j8djg\""
Sep 17 14:34:35.101606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.101582    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"]
Sep 17 14:34:35.128177 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.128148    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:35.267095 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-certs\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.267301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-podinfo\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.267301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-socket\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.267301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267242    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-token\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.267438 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istiod-ca-cert\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.267438 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-credential-socket\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.267517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fsbx\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-kube-api-access-9fsbx\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.267517 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-envoy\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.267605 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.267517    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-data\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.271005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.270981    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:35.368838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.368800    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-token\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.368913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istiod-ca-cert\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.368955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-credential-socket\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369030 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.368986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fsbx\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-kube-api-access-9fsbx\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-envoy\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369065    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-data\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369102    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-certs\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369160    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-podinfo\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-socket\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369508 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-credential-socket\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-data\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-socket\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369653    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-certs\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.369788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.369767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istiod-ca-cert\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.371423 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.371400    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-envoy\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.371622 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.371602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-podinfo\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.372854 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.372830    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:35.376750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.376720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-token\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.376931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.376915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fsbx\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-kube-api-access-9fsbx\") pod \"e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") " pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.396450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.396433    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:35.682642 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.682612    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"]
Sep 17 14:34:35.683501 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:34:35.683467    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e2acd3_48ba_462f_a17a_7c0ca48427ed.slice/crio-0ec5e964e62e385913da9afdeba47bc31633bb2da1184ca4e1b050da09233b85 WatchSource:0}: Error finding container 0ec5e964e62e385913da9afdeba47bc31633bb2da1184ca4e1b050da09233b85: Status 404 returned error can't find the container with id 0ec5e964e62e385913da9afdeba47bc31633bb2da1184ca4e1b050da09233b85
Sep 17 14:34:35.685919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.685886    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:34:35.686017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.685965    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:34:35.686017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.685993    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:34:35.878647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.878605    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" event={"ID":"d9e2acd3-48ba-462f-a17a-7c0ca48427ed","Type":"ContainerStarted","Data":"03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35"}
Sep 17 14:34:35.879118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.878702    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" event={"ID":"d9e2acd3-48ba-462f-a17a-7c0ca48427ed","Type":"ContainerStarted","Data":"0ec5e964e62e385913da9afdeba47bc31633bb2da1184ca4e1b050da09233b85"}
Sep 17 14:34:35.902651 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:35.902604    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" podStartSLOduration=0.902587126 podStartE2EDuration="902.587126ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:34:35.900297253 +0000 UTC m=+1662.833802445" watchObservedRunningTime="2026-09-17 14:34:35.902587126 +0000 UTC m=+1662.836092318"
Sep 17 14:34:36.098817 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:36.098781    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:36.397322 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:36.397217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:36.402958 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:36.402931    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:36.885072 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:36.885041    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:36.885964 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:36.885927    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:34:37.255914 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:37.255825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:37.871808 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:37.871764    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:37.963641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:37.963608    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:38.064161 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:38.064127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:38.948514 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:38.948476    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:39.769487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:39.769456    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:40.737616 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:40.737575    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:41.433717 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:41.433682    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:42.254834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:42.254796    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:43.256377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.256341    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:43.355723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.355693    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:43.719476 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.719424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:43.812988 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.812950    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:43.813489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.813467    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"]
Sep 17 14:34:43.818288 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.818243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:43.821583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.821560    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0829c1\""
Sep 17 14:34:43.828783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.828739    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"]
Sep 17 14:34:43.863939 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.863901    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7\" (UID: \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:43.864119 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.864075    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5k7\" (UniqueName: \"kubernetes.io/projected/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-kube-api-access-qd5k7\") pod \"payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7\" (UID: \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:43.967287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.966478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5k7\" (UniqueName: \"kubernetes.io/projected/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-kube-api-access-qd5k7\") pod \"payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7\" (UID: \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:43.967287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.966618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7\" (UID: \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:43.967619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.967582    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7\" (UID: \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:43.983545 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:43.982789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5k7\" (UniqueName: \"kubernetes.io/projected/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-kube-api-access-qd5k7\") pod \"payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7\" (UID: \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:44.065706 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.065667    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:44.130782 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.130745    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:44.174793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.174213    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf" podUID="bb2e1881-ce05-467b-be0c-5dc0d27b899f" containerName="main" containerID="cri-o://5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a" gracePeriod=45
Sep 17 14:34:44.207047 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.206921    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:44.353076 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.353000    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:44.417986 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.417879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"]
Sep 17 14:34:44.418898 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:34:44.418824    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6aa982b_a31a_4763_9d06_9a6c15e9fe95.slice/crio-583dabcafab34622b33f52a232c3ee33193ace5379b6dff8bff4c31feba6b057 WatchSource:0}: Error finding container 583dabcafab34622b33f52a232c3ee33193ace5379b6dff8bff4c31feba6b057: Status 404 returned error can't find the container with id 583dabcafab34622b33f52a232c3ee33193ace5379b6dff8bff4c31feba6b057
Sep 17 14:34:44.437284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.432013    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:44.437284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.432075    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"]
Sep 17 14:34:44.437284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.435843    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-080801-qcbj5"]
Sep 17 14:34:44.488661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.488606    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:34:44.579056 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579022    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-model-cache\") pod \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") "
Sep 17 14:34:44.579244 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579090    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-home\") pod \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") "
Sep 17 14:34:44.579244 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579127    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-dshm\") pod \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") "
Sep 17 14:34:44.579244 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-model-cache" (OuterVolumeSpecName: "model-cache") pod "bb2e1881-ce05-467b-be0c-5dc0d27b899f" (UID: "bb2e1881-ce05-467b-be0c-5dc0d27b899f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:44.579244 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tls-certs\") pod \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") "
Sep 17 14:34:44.579481 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579289    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w47kj\" (UniqueName: \"kubernetes.io/projected/bb2e1881-ce05-467b-be0c-5dc0d27b899f-kube-api-access-w47kj\") pod \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") "
Sep 17 14:34:44.579481 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579331    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tmp-dir\") pod \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\" (UID: \"bb2e1881-ce05-467b-be0c-5dc0d27b899f\") "
Sep 17 14:34:44.579481 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579410    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-home" (OuterVolumeSpecName: "home") pod "bb2e1881-ce05-467b-be0c-5dc0d27b899f" (UID: "bb2e1881-ce05-467b-be0c-5dc0d27b899f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:44.579682 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579655    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bb2e1881-ce05-467b-be0c-5dc0d27b899f" (UID: "bb2e1881-ce05-467b-be0c-5dc0d27b899f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:44.580020 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.579994    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.580169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.580130    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.580169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.580158    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.581802 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.581770    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2e1881-ce05-467b-be0c-5dc0d27b899f-kube-api-access-w47kj" (OuterVolumeSpecName: "kube-api-access-w47kj") pod "bb2e1881-ce05-467b-be0c-5dc0d27b899f" (UID: "bb2e1881-ce05-467b-be0c-5dc0d27b899f"). InnerVolumeSpecName "kube-api-access-w47kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:34:44.581802 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.581782    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-dshm" (OuterVolumeSpecName: "dshm") pod "bb2e1881-ce05-467b-be0c-5dc0d27b899f" (UID: "bb2e1881-ce05-467b-be0c-5dc0d27b899f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:44.581936 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.581905    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bb2e1881-ce05-467b-be0c-5dc0d27b899f" (UID: "bb2e1881-ce05-467b-be0c-5dc0d27b899f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:34:44.681557 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.681523    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb2e1881-ce05-467b-be0c-5dc0d27b899f-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.681557 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.681554    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w47kj\" (UniqueName: \"kubernetes.io/projected/bb2e1881-ce05-467b-be0c-5dc0d27b899f-kube-api-access-w47kj\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.681773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.681568    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb2e1881-ce05-467b-be0c-5dc0d27b899f-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:44.785412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.785377    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:44.935208 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.935126    2565 generic.go:356] "Generic (PLEG): container finished" podID="bb2e1881-ce05-467b-be0c-5dc0d27b899f" containerID="5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a" exitCode=0
Sep 17 14:34:44.935208 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.935167    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"
Sep 17 14:34:44.935450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.935209    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf" event={"ID":"bb2e1881-ce05-467b-be0c-5dc0d27b899f","Type":"ContainerDied","Data":"5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a"}
Sep 17 14:34:44.935450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.935270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf" event={"ID":"bb2e1881-ce05-467b-be0c-5dc0d27b899f","Type":"ContainerDied","Data":"2867dc01368c750cb62645146564e1424077d2caf60f3aa3130bbb8539061465"}
Sep 17 14:34:44.935450 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.935292    2565 scope.go:117] "RemoveContainer" containerID="5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a"
Sep 17 14:34:44.937115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.937089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7" event={"ID":"a6aa982b-a31a-4763-9d06-9a6c15e9fe95","Type":"ContainerStarted","Data":"b3f9cd1bc8d7e335640cf52f0d7712c9c4634f414dee3b0c04f8e0d6e1a45852"}
Sep 17 14:34:44.937230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.937139    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7" event={"ID":"a6aa982b-a31a-4763-9d06-9a6c15e9fe95","Type":"ContainerStarted","Data":"583dabcafab34622b33f52a232c3ee33193ace5379b6dff8bff4c31feba6b057"}
Sep 17 14:34:44.937230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.937189    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:44.947490 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.947469    2565 scope.go:117] "RemoveContainer" containerID="5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a"
Sep 17 14:34:44.947769 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:34:44.947751    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a\": container with ID starting with 5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a not found: ID does not exist" containerID="5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a"
Sep 17 14:34:44.947819 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.947778    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a"} err="failed to get container status \"5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a\": rpc error: code = NotFound desc = could not find container \"5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a\": container with ID starting with 5cf44fe1cd8f556687cc11ae1c2f8bd0776216a3f30e54fb5de2e3fc4a795a0a not found: ID does not exist"
Sep 17 14:34:44.962007 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.961944    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7" podStartSLOduration=1.961928929 podStartE2EDuration="1.961928929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:34:44.958983679 +0000 UTC m=+1671.892488870" watchObservedRunningTime="2026-09-17 14:34:44.961928929 +0000 UTC m=+1671.895434121"
Sep 17 14:34:44.983289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.983246    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"]
Sep 17 14:34:44.989758 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:44.989729    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-080801/distinct-w0-080801-kserve-9cbdb9576-zd6cf"]
Sep 17 14:34:45.244807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:45.244691    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:45.521842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:45.521766    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2e1881-ce05-467b-be0c-5dc0d27b899f" path="/var/lib/kubelet/pods/bb2e1881-ce05-467b-be0c-5dc0d27b899f/volumes"
Sep 17 14:34:45.522215 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:45.522109    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06fe340-d231-4fcd-ae67-47e7c4ac9781" path="/var/lib/kubelet/pods/d06fe340-d231-4fcd-ae67-47e7c4ac9781/volumes"
Sep 17 14:34:46.029697 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:46.029669    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:46.816494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:46.816465    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:47.804787 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:47.804736    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:48.646006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:48.645963    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:49.567214 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:49.567178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:50.353919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:50.353882    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:51.298585 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:51.298549    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:51.770739 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:51.770709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:51.907437 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:51.907403    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:52.042612 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:52.042531    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"]
Sep 17 14:34:52.042761 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:52.042740    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" containerID="cri-o://348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553" gracePeriod=30
Sep 17 14:34:52.113537 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:52.113498    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:52.684176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:52.684144    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:53.120878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:53.120842    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:53.892297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:53.892248    2565 scope.go:117] "RemoveContainer" containerID="9fe26853a0695d86ee06ee292808110338aaa8e34fbb15d10e3059d5600cf777"
Sep 17 14:34:54.043824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:54.043774    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:54.494940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:54.494905    2565 patch_prober.go:28] interesting pod/e2e-worker-w0-080801-openshift-default-84855b7986-dx695 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:34:54.495401 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:54.494975    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:34:54.725627 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:54.725581    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:55.480800 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.480768    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"]
Sep 17 14:34:55.481290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.481276    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2e1881-ce05-467b-be0c-5dc0d27b899f" containerName="main"
Sep 17 14:34:55.481341 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.481291    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2e1881-ce05-467b-be0c-5dc0d27b899f" containerName="main"
Sep 17 14:34:55.481392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.481383    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2e1881-ce05-467b-be0c-5dc0d27b899f" containerName="main"
Sep 17 14:34:55.484409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.484391    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"
Sep 17 14:34:55.486879 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.486862    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:34:55.486879 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.486871    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:34:55.495905 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.495884    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"]
Sep 17 14:34:55.614559 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.614525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/facd1ce1-9154-41af-9417-d0653325143c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29d3c7-664sx\" (UID: \"facd1ce1-9154-41af-9417-d0653325143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"
Sep 17 14:34:55.715542 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.715506    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/facd1ce1-9154-41af-9417-d0653325143c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29d3c7-664sx\" (UID: \"facd1ce1-9154-41af-9417-d0653325143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"
Sep 17 14:34:55.716136 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.716112    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/facd1ce1-9154-41af-9417-d0653325143c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-29d3c7-664sx\" (UID: \"facd1ce1-9154-41af-9417-d0653325143c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"
Sep 17 14:34:55.795885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.795799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"
Sep 17 14:34:55.842031 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.841974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:55.946757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:55.946728    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:34:56.074461 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:56.074429    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"]
Sep 17 14:34:56.076217 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:34:56.076190    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfacd1ce1_9154_41af_9417_d0653325143c.slice/crio-ea673fc1b4018ae29781c84e02ea9df9c15d84276b9d7b2aa57239fff6c7e4bd WatchSource:0}: Error finding container ea673fc1b4018ae29781c84e02ea9df9c15d84276b9d7b2aa57239fff6c7e4bd: Status 404 returned error can't find the container with id ea673fc1b4018ae29781c84e02ea9df9c15d84276b9d7b2aa57239fff6c7e4bd
Sep 17 14:34:56.759723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:56.759694    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:57.017101 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.017023    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerID="4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b" exitCode=137
Sep 17 14:34:57.017273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.017096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz" event={"ID":"0c4317e5-a5c3-4e14-b889-0b45c3e4e082","Type":"ContainerDied","Data":"4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b"}
Sep 17 14:34:57.017542 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.017516    2565 scope.go:117] "RemoveContainer" containerID="4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b"
Sep 17 14:34:57.018939 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.018914    2565 generic.go:356] "Generic (PLEG): container finished" podID="facd1ce1-9154-41af-9417-d0653325143c" containerID="02509ff452fbd8c00c6bd53a921d874daf8616519622ab4003bfc60914a1bbda" exitCode=0
Sep 17 14:34:57.019048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.018959    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx" event={"ID":"facd1ce1-9154-41af-9417-d0653325143c","Type":"ContainerDied","Data":"02509ff452fbd8c00c6bd53a921d874daf8616519622ab4003bfc60914a1bbda"}
Sep 17 14:34:57.019048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.018996    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx" event={"ID":"facd1ce1-9154-41af-9417-d0653325143c","Type":"ContainerStarted","Data":"ea673fc1b4018ae29781c84e02ea9df9c15d84276b9d7b2aa57239fff6c7e4bd"}
Sep 17 14:34:57.314812 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.314788    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:34:57.433975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.433938    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-data\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.433988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-credential-socket\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-envoy\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-token\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434123    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-podinfo\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istiod-ca-cert\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434300    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434312    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-certs\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434390    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxqqq\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-kube-api-access-pxqqq\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434416    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-socket\") pod \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\" (UID: \"309c5898-acbe-4ad6-ba82-c3a9c51c26c3\") "
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434490    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434592    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434731    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434923    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434942    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434954    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434966    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:57.436388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.434975    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:57.437052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.436447    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:34:57.437052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.436477    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:34:57.437052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.436712    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:34:57.437052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.436763    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-kube-api-access-pxqqq" (OuterVolumeSpecName: "kube-api-access-pxqqq") pod "309c5898-acbe-4ad6-ba82-c3a9c51c26c3" (UID: "309c5898-acbe-4ad6-ba82-c3a9c51c26c3"). InnerVolumeSpecName "kube-api-access-pxqqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:34:57.462373 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.462345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:57.536479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.536451    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:57.536479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.536478    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxqqq\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-kube-api-access-pxqqq\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:57.536630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.536487    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:57.536630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:57.536495    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/309c5898-acbe-4ad6-ba82-c3a9c51c26c3-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:58.027487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.027448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz" event={"ID":"0c4317e5-a5c3-4e14-b889-0b45c3e4e082","Type":"ContainerStarted","Data":"32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb"}
Sep 17 14:34:58.027970 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.027661    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:34:58.029229 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.029202    2565 generic.go:356] "Generic (PLEG): container finished" podID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerID="348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553" exitCode=0
Sep 17 14:34:58.029365 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.029276    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"
Sep 17 14:34:58.029365 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.029281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" event={"ID":"309c5898-acbe-4ad6-ba82-c3a9c51c26c3","Type":"ContainerDied","Data":"348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553"}
Sep 17 14:34:58.029365 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.029334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695" event={"ID":"309c5898-acbe-4ad6-ba82-c3a9c51c26c3","Type":"ContainerDied","Data":"64880d299304641246bb8313315ff6641bfb2bd465ae69c55ecbecd430ecc113"}
Sep 17 14:34:58.029476 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.029365    2565 scope.go:117] "RemoveContainer" containerID="348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553"
Sep 17 14:34:58.039435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.039415    2565 scope.go:117] "RemoveContainer" containerID="c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712"
Sep 17 14:34:58.050282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.049732    2565 scope.go:117] "RemoveContainer" containerID="348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553"
Sep 17 14:34:58.050906 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:34:58.050511    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553\": container with ID starting with 348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553 not found: ID does not exist" containerID="348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553"
Sep 17 14:34:58.050906 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.050549    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553"} err="failed to get container status \"348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553\": rpc error: code = NotFound desc = could not find container \"348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553\": container with ID starting with 348eda92cc8b65336743f5606cc9b892c1f90103cf0554cc2117ec0951ab1553 not found: ID does not exist"
Sep 17 14:34:58.050906 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.050583    2565 scope.go:117] "RemoveContainer" containerID="c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712"
Sep 17 14:34:58.051174 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:34:58.050974    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712\": container with ID starting with c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712 not found: ID does not exist" containerID="c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712"
Sep 17 14:34:58.051174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.051035    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712"} err="failed to get container status \"c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712\": rpc error: code = NotFound desc = could not find container \"c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712\": container with ID starting with c944df482c42fe5c8389c908ce9654824953e460f233347f16ab4ea3863c3712 not found: ID does not exist"
Sep 17 14:34:58.070781 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.070747    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"]
Sep 17 14:34:58.073350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.073327    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-080801-openshift-default-84855b7986-dx695"]
Sep 17 14:34:58.195535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.195478    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"
Sep 17 14:34:58.345681 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.345642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/facd1ce1-9154-41af-9417-d0653325143c-maas-api-service-ca\") pod \"facd1ce1-9154-41af-9417-d0653325143c\" (UID: \"facd1ce1-9154-41af-9417-d0653325143c\") "
Sep 17 14:34:58.346004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.345982    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/facd1ce1-9154-41af-9417-d0653325143c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "facd1ce1-9154-41af-9417-d0653325143c" (UID: "facd1ce1-9154-41af-9417-d0653325143c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:34:58.346105 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.346090    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/facd1ce1-9154-41af-9417-d0653325143c-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:34:58.353147 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:58.353106    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:59.037711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:59.037680    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"
Sep 17 14:34:59.038176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:59.037723    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx" event={"ID":"facd1ce1-9154-41af-9417-d0653325143c","Type":"ContainerDied","Data":"ea673fc1b4018ae29781c84e02ea9df9c15d84276b9d7b2aa57239fff6c7e4bd"}
Sep 17 14:34:59.038176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:59.037775    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea673fc1b4018ae29781c84e02ea9df9c15d84276b9d7b2aa57239fff6c7e4bd"
Sep 17 14:34:59.347273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:59.347231    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:34:59.521134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:34:59.521103    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" path="/var/lib/kubelet/pods/309c5898-acbe-4ad6-ba82-c3a9c51c26c3/volumes"
Sep 17 14:35:00.302240 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:00.302205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:01.037897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:01.037862    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:02.077921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:02.077889    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:02.823200 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:02.823166    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:04.143935 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:04.143905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:06.702028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:06.701980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:06.780690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:06.780663    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:07.204488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:07.204457    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:08.438519 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.438492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:08.468273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.468212    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"]
Sep 17 14:35:08.468520 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.468495    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs" podUID="e8643c32-3647-4c36-90f6-c2896ec66f08" containerName="payload-processing" containerID="cri-o://2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb" gracePeriod=30
Sep 17 14:35:08.473088 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.473065    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"]
Sep 17 14:35:08.473427 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.473403    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f" podUID="7459ff24-4a73-484b-a562-720925fed82b" containerName="payload-pre-processing" containerID="cri-o://c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e" gracePeriod=30
Sep 17 14:35:08.545660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.545627    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:08.646397 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.646325    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:08.675048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.675019    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"]
Sep 17 14:35:08.680727 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.680691    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-29d3c7-664sx"]
Sep 17 14:35:08.740154 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.740093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:08.765090 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.765070    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:35:08.768264 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.768236    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:35:08.862485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.862449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8j97t\" (UniqueName: \"kubernetes.io/projected/e8643c32-3647-4c36-90f6-c2896ec66f08-kube-api-access-8j97t\") pod \"e8643c32-3647-4c36-90f6-c2896ec66f08\" (UID: \"e8643c32-3647-4c36-90f6-c2896ec66f08\") "
Sep 17 14:35:08.862659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.862518    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8643c32-3647-4c36-90f6-c2896ec66f08-plugins-config-volume\") pod \"e8643c32-3647-4c36-90f6-c2896ec66f08\" (UID: \"e8643c32-3647-4c36-90f6-c2896ec66f08\") "
Sep 17 14:35:08.862659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.862550    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ff24-4a73-484b-a562-720925fed82b-plugins-config-volume\") pod \"7459ff24-4a73-484b-a562-720925fed82b\" (UID: \"7459ff24-4a73-484b-a562-720925fed82b\") "
Sep 17 14:35:08.862659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.862644    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-997nf\" (UniqueName: \"kubernetes.io/projected/7459ff24-4a73-484b-a562-720925fed82b-kube-api-access-997nf\") pod \"7459ff24-4a73-484b-a562-720925fed82b\" (UID: \"7459ff24-4a73-484b-a562-720925fed82b\") "
Sep 17 14:35:08.862924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.862901    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7459ff24-4a73-484b-a562-720925fed82b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7459ff24-4a73-484b-a562-720925fed82b" (UID: "7459ff24-4a73-484b-a562-720925fed82b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:08.863041 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.862941    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8643c32-3647-4c36-90f6-c2896ec66f08-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8643c32-3647-4c36-90f6-c2896ec66f08" (UID: "e8643c32-3647-4c36-90f6-c2896ec66f08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:08.863170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.863154    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8643c32-3647-4c36-90f6-c2896ec66f08-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:08.863218 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.863172    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7459ff24-4a73-484b-a562-720925fed82b-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:08.864664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.864643    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8643c32-3647-4c36-90f6-c2896ec66f08-kube-api-access-8j97t" (OuterVolumeSpecName: "kube-api-access-8j97t") pod "e8643c32-3647-4c36-90f6-c2896ec66f08" (UID: "e8643c32-3647-4c36-90f6-c2896ec66f08"). InnerVolumeSpecName "kube-api-access-8j97t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:08.864736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.864672    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7459ff24-4a73-484b-a562-720925fed82b-kube-api-access-997nf" (OuterVolumeSpecName: "kube-api-access-997nf") pod "7459ff24-4a73-484b-a562-720925fed82b" (UID: "7459ff24-4a73-484b-a562-720925fed82b"). InnerVolumeSpecName "kube-api-access-997nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:08.964623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.964579    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j97t\" (UniqueName: \"kubernetes.io/projected/e8643c32-3647-4c36-90f6-c2896ec66f08-kube-api-access-8j97t\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:08.964623 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:08.964618    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-997nf\" (UniqueName: \"kubernetes.io/projected/7459ff24-4a73-484b-a562-720925fed82b-kube-api-access-997nf\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:09.020536 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.020424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:09.039850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.039815    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:35:09.101469 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.101431    2565 generic.go:356] "Generic (PLEG): container finished" podID="7459ff24-4a73-484b-a562-720925fed82b" containerID="c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e" exitCode=0
Sep 17 14:35:09.101469 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.101477    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"
Sep 17 14:35:09.101745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.101512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f" event={"ID":"7459ff24-4a73-484b-a562-720925fed82b","Type":"ContainerDied","Data":"c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e"}
Sep 17 14:35:09.101745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.101580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f" event={"ID":"7459ff24-4a73-484b-a562-720925fed82b","Type":"ContainerDied","Data":"5fb0ef1fb265ed54b7ed708d6bc19edb5add6916b59209c74bb406c53d16db3a"}
Sep 17 14:35:09.101745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.101623    2565 scope.go:117] "RemoveContainer" containerID="c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e"
Sep 17 14:35:09.103957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.103905    2565 generic.go:356] "Generic (PLEG): container finished" podID="e8643c32-3647-4c36-90f6-c2896ec66f08" containerID="2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb" exitCode=0
Sep 17 14:35:09.103957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.103971    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"
Sep 17 14:35:09.104266 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.104098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs" event={"ID":"e8643c32-3647-4c36-90f6-c2896ec66f08","Type":"ContainerDied","Data":"2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb"}
Sep 17 14:35:09.104266 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.104160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs" event={"ID":"e8643c32-3647-4c36-90f6-c2896ec66f08","Type":"ContainerDied","Data":"ce5ac42eb0e805514b6a86a7d238c5d58f9dcac5d158dfd714e908ed01aaece7"}
Sep 17 14:35:09.114700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.114681    2565 scope.go:117] "RemoveContainer" containerID="c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e"
Sep 17 14:35:09.115019 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:35:09.114994    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e\": container with ID starting with c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e not found: ID does not exist" containerID="c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e"
Sep 17 14:35:09.115112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.115032    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e"} err="failed to get container status \"c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e\": rpc error: code = NotFound desc = could not find container \"c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e\": container with ID starting with c281ce11e77a8de3a9d77387d1ed78e4eb4351737078011ec624c43c37f5842e not found: ID does not exist"
Sep 17 14:35:09.115112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.115057    2565 scope.go:117] "RemoveContainer" containerID="2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb"
Sep 17 14:35:09.124425 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.124409    2565 scope.go:117] "RemoveContainer" containerID="2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb"
Sep 17 14:35:09.124691 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:35:09.124673    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb\": container with ID starting with 2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb not found: ID does not exist" containerID="2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb"
Sep 17 14:35:09.124779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.124697    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb"} err="failed to get container status \"2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb\": rpc error: code = NotFound desc = could not find container \"2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb\": container with ID starting with 2af54ec390b719eac28f968d7ee04c5743e0fbf240ddec313ba064dd4ef184fb not found: ID does not exist"
Sep 17 14:35:09.139966 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.139944    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"]
Sep 17 14:35:09.148634 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.148606    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-29d3c7-5d5bc47d7f-2bnfs"]
Sep 17 14:35:09.165747 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.165722    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:09.167084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.167064    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"]
Sep 17 14:35:09.171041 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.171021    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-29d3c7-6bb79dbd74-8486f"]
Sep 17 14:35:09.520613 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.520583    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7459ff24-4a73-484b-a562-720925fed82b" path="/var/lib/kubelet/pods/7459ff24-4a73-484b-a562-720925fed82b/volumes"
Sep 17 14:35:09.521003 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.520988    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8643c32-3647-4c36-90f6-c2896ec66f08" path="/var/lib/kubelet/pods/e8643c32-3647-4c36-90f6-c2896ec66f08/volumes"
Sep 17 14:35:09.521326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:09.521313    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="facd1ce1-9154-41af-9417-d0653325143c" path="/var/lib/kubelet/pods/facd1ce1-9154-41af-9417-d0653325143c/volumes"
Sep 17 14:35:10.142473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:10.142444    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:10.948357 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:10.948317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:11.857157 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:11.857124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:12.711522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:12.711473    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:13.385642 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:13.385608    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:13.487635 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:13.487599    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:14.566695 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:14.566652    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:15.644067 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:15.644032    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:16.775942 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:16.775855    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:16.851827 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:16.851784    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:17.049715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.049613    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:17.370625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.370589    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"]
Sep 17 14:35:17.371042 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371030    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy"
Sep 17 14:35:17.371089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371044    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy"
Sep 17 14:35:17.371089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371073    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8643c32-3647-4c36-90f6-c2896ec66f08" containerName="payload-processing"
Sep 17 14:35:17.371089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371080    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8643c32-3647-4c36-90f6-c2896ec66f08" containerName="payload-processing"
Sep 17 14:35:17.371089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371088    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="facd1ce1-9154-41af-9417-d0653325143c" containerName="revoke-keys"
Sep 17 14:35:17.371221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371094    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="facd1ce1-9154-41af-9417-d0653325143c" containerName="revoke-keys"
Sep 17 14:35:17.371221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371107    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7459ff24-4a73-484b-a562-720925fed82b" containerName="payload-pre-processing"
Sep 17 14:35:17.371221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371112    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7459ff24-4a73-484b-a562-720925fed82b" containerName="payload-pre-processing"
Sep 17 14:35:17.371221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371175    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8643c32-3647-4c36-90f6-c2896ec66f08" containerName="payload-processing"
Sep 17 14:35:17.371221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371184    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7459ff24-4a73-484b-a562-720925fed82b" containerName="payload-pre-processing"
Sep 17 14:35:17.371221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371192    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="facd1ce1-9154-41af-9417-d0653325143c" containerName="revoke-keys"
Sep 17 14:35:17.371221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371199    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy"
Sep 17 14:35:17.371221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.371207    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy"
Sep 17 14:35:17.375952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.375931    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"
Sep 17 14:35:17.378321 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.378245    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:35:17.378454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.378319    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:35:17.382616 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.382590    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"]
Sep 17 14:35:17.459608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.459569    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c56ddd-df18-44b2-a45b-7ba3cb7a292a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0829c1-vmt29\" (UID: \"15c56ddd-df18-44b2-a45b-7ba3cb7a292a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"
Sep 17 14:35:17.561066 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.561023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c56ddd-df18-44b2-a45b-7ba3cb7a292a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0829c1-vmt29\" (UID: \"15c56ddd-df18-44b2-a45b-7ba3cb7a292a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"
Sep 17 14:35:17.561800 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.561777    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c56ddd-df18-44b2-a45b-7ba3cb7a292a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-0829c1-vmt29\" (UID: \"15c56ddd-df18-44b2-a45b-7ba3cb7a292a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"
Sep 17 14:35:17.646033 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.645945    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:17.689910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.689876    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"
Sep 17 14:35:17.978838 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:17.978808    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"]
Sep 17 14:35:17.980174 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:35:17.980141    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15c56ddd_df18_44b2_a45b_7ba3cb7a292a.slice/crio-dd4559851acd952e712496db4f9ac35bab035c421faea03382d31cdc4b8889cd WatchSource:0}: Error finding container dd4559851acd952e712496db4f9ac35bab035c421faea03382d31cdc4b8889cd: Status 404 returned error can't find the container with id dd4559851acd952e712496db4f9ac35bab035c421faea03382d31cdc4b8889cd
Sep 17 14:35:18.170132 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:18.170097    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29" event={"ID":"15c56ddd-df18-44b2-a45b-7ba3cb7a292a","Type":"ContainerStarted","Data":"dd4559851acd952e712496db4f9ac35bab035c421faea03382d31cdc4b8889cd"}
Sep 17 14:35:18.534867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:18.534742    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:19.177866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:19.177833    2565 generic.go:356] "Generic (PLEG): container finished" podID="15c56ddd-df18-44b2-a45b-7ba3cb7a292a" containerID="a0e9811618aefd44d1e6d29f93490d48b8094a26bf1f5f213479da18dd93667c" exitCode=0
Sep 17 14:35:19.178237 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:19.177927    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29" event={"ID":"15c56ddd-df18-44b2-a45b-7ba3cb7a292a","Type":"ContainerDied","Data":"a0e9811618aefd44d1e6d29f93490d48b8094a26bf1f5f213479da18dd93667c"}
Sep 17 14:35:19.327202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:19.327120    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:20.138018 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:20.137987    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:20.327658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:20.327625    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"
Sep 17 14:35:20.393398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:20.393315    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c56ddd-df18-44b2-a45b-7ba3cb7a292a-maas-api-service-ca\") pod \"15c56ddd-df18-44b2-a45b-7ba3cb7a292a\" (UID: \"15c56ddd-df18-44b2-a45b-7ba3cb7a292a\") "
Sep 17 14:35:20.393748 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:20.393723    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15c56ddd-df18-44b2-a45b-7ba3cb7a292a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "15c56ddd-df18-44b2-a45b-7ba3cb7a292a" (UID: "15c56ddd-df18-44b2-a45b-7ba3cb7a292a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:20.393955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:20.393939    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15c56ddd-df18-44b2-a45b-7ba3cb7a292a-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:20.819413 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:20.819328    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:21.194006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:21.193977    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"
Sep 17 14:35:21.194006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:21.193983    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29" event={"ID":"15c56ddd-df18-44b2-a45b-7ba3cb7a292a","Type":"ContainerDied","Data":"dd4559851acd952e712496db4f9ac35bab035c421faea03382d31cdc4b8889cd"}
Sep 17 14:35:21.194225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:21.194031    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd4559851acd952e712496db4f9ac35bab035c421faea03382d31cdc4b8889cd"
Sep 17 14:35:21.497981 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:21.497901    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:22.374771 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:22.374727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:23.147689 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:23.147640    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:26.366219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:26.366178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:26.474924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:26.474881    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:28.529955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:28.529925    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:32.948559 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:32.948427    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:32.986972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:32.986925    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"]
Sep 17 14:35:32.987842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:32.987812    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7" podUID="a6aa982b-a31a-4763-9d06-9a6c15e9fe95" containerName="payload-pre-processing" containerID="cri-o://b3f9cd1bc8d7e335640cf52f0d7712c9c4634f414dee3b0c04f8e0d6e1a45852" gracePeriod=30
Sep 17 14:35:33.019754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.019726    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:33.122442 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.122392    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:33.130955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.130907    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"]
Sep 17 14:35:33.134907 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.134868    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-0829c1-vmt29"]
Sep 17 14:35:33.180454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.180381    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:33.280793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.280758    2565 generic.go:356] "Generic (PLEG): container finished" podID="a6aa982b-a31a-4763-9d06-9a6c15e9fe95" containerID="b3f9cd1bc8d7e335640cf52f0d7712c9c4634f414dee3b0c04f8e0d6e1a45852" exitCode=0
Sep 17 14:35:33.280975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.280831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7" event={"ID":"a6aa982b-a31a-4763-9d06-9a6c15e9fe95","Type":"ContainerDied","Data":"b3f9cd1bc8d7e335640cf52f0d7712c9c4634f414dee3b0c04f8e0d6e1a45852"}
Sep 17 14:35:33.280975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.280893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7" event={"ID":"a6aa982b-a31a-4763-9d06-9a6c15e9fe95","Type":"ContainerDied","Data":"583dabcafab34622b33f52a232c3ee33193ace5379b6dff8bff4c31feba6b057"}
Sep 17 14:35:33.280975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.280918    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="583dabcafab34622b33f52a232c3ee33193ace5379b6dff8bff4c31feba6b057"
Sep 17 14:35:33.284585 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.284557    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:35:33.339367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.339330    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qd5k7\" (UniqueName: \"kubernetes.io/projected/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-kube-api-access-qd5k7\") pod \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\" (UID: \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\") "
Sep 17 14:35:33.339567 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.339423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-plugins-config-volume\") pod \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\" (UID: \"a6aa982b-a31a-4763-9d06-9a6c15e9fe95\") "
Sep 17 14:35:33.339925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.339894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a6aa982b-a31a-4763-9d06-9a6c15e9fe95" (UID: "a6aa982b-a31a-4763-9d06-9a6c15e9fe95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:33.341629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.341608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-kube-api-access-qd5k7" (OuterVolumeSpecName: "kube-api-access-qd5k7") pod "a6aa982b-a31a-4763-9d06-9a6c15e9fe95" (UID: "a6aa982b-a31a-4763-9d06-9a6c15e9fe95"). InnerVolumeSpecName "kube-api-access-qd5k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:33.440729 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.440680    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd5k7\" (UniqueName: \"kubernetes.io/projected/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-kube-api-access-qd5k7\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:33.440729 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.440722    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6aa982b-a31a-4763-9d06-9a6c15e9fe95-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:33.478529 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.478484    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:33.522273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.522175    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15c56ddd-df18-44b2-a45b-7ba3cb7a292a" path="/var/lib/kubelet/pods/15c56ddd-df18-44b2-a45b-7ba3cb7a292a/volumes"
Sep 17 14:35:33.616821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:33.616792    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:34.240102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:34.240071    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:34.286380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:34.286349    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"
Sep 17 14:35:34.307880 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:34.307812    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"]
Sep 17 14:35:34.311633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:34.311584    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-0829c1-fdf65b55d-9sbf7"]
Sep 17 14:35:34.927246 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:34.927211    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:35.520677 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:35.520602    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6aa982b-a31a-4763-9d06-9a6c15e9fe95" path="/var/lib/kubelet/pods/a6aa982b-a31a-4763-9d06-9a6c15e9fe95/volumes"
Sep 17 14:35:35.741444 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:35.741347    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:36.301621 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:36.301590    2565 generic.go:356] "Generic (PLEG): container finished" podID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerID="03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35" exitCode=0
Sep 17 14:35:36.301819 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:36.301645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" event={"ID":"d9e2acd3-48ba-462f-a17a-7c0ca48427ed","Type":"ContainerDied","Data":"03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35"}
Sep 17 14:35:36.302118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:36.302102    2565 scope.go:117] "RemoveContainer" containerID="03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35"
Sep 17 14:35:36.303404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:36.303377    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:35:36.303470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:36.303445    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:35:36.303506 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:36.303475    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:35:36.534245 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:36.534206    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:36.885500 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:36.885461    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:35:37.312430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:37.312330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" event={"ID":"d9e2acd3-48ba-462f-a17a-7c0ca48427ed","Type":"ContainerStarted","Data":"d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d"}
Sep 17 14:35:37.341836 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:37.341801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:37.397444 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:37.397404    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:35:37.398712 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:37.398682    2565 patch_prober.go:28] interesting pod/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 17 14:35:37.398866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:37.398741    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 17 14:35:38.396953 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:38.396912    2565 patch_prober.go:28] interesting pod/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 17 14:35:38.397389 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:38.396978    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 17 14:35:39.252649 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.251277    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:39.353867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.353829    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:39.396867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.396824    2565 patch_prober.go:28] interesting pod/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Sep 17 14:35:39.397041 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.396903    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Sep 17 14:35:39.493856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.493820    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"]
Sep 17 14:35:39.496442 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.496408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:39.820089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820051    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"]
Sep 17 14:35:39.820610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820594    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy"
Sep 17 14:35:39.820665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820620    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="309c5898-acbe-4ad6-ba82-c3a9c51c26c3" containerName="istio-proxy"
Sep 17 14:35:39.820665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820629    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6aa982b-a31a-4763-9d06-9a6c15e9fe95" containerName="payload-pre-processing"
Sep 17 14:35:39.820665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820654    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6aa982b-a31a-4763-9d06-9a6c15e9fe95" containerName="payload-pre-processing"
Sep 17 14:35:39.820786 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820680    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15c56ddd-df18-44b2-a45b-7ba3cb7a292a" containerName="revoke-keys"
Sep 17 14:35:39.820786 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820686    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="15c56ddd-df18-44b2-a45b-7ba3cb7a292a" containerName="revoke-keys"
Sep 17 14:35:39.820786 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820773    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="15c56ddd-df18-44b2-a45b-7ba3cb7a292a" containerName="revoke-keys"
Sep 17 14:35:39.820786 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.820781    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6aa982b-a31a-4763-9d06-9a6c15e9fe95" containerName="payload-pre-processing"
Sep 17 14:35:39.823930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.823913    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"
Sep 17 14:35:39.826876 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.826853    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:35:39.827144 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.827123    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:35:39.834882 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.834856    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"]
Sep 17 14:35:39.912350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:39.912308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86598416-2866-4dd3-b9a4-28525899abda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b\" (UID: \"86598416-2866-4dd3-b9a4-28525899abda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"
Sep 17 14:35:40.013738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:40.013703    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86598416-2866-4dd3-b9a4-28525899abda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b\" (UID: \"86598416-2866-4dd3-b9a4-28525899abda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"
Sep 17 14:35:40.014380 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:40.014358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86598416-2866-4dd3-b9a4-28525899abda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b\" (UID: \"86598416-2866-4dd3-b9a4-28525899abda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"
Sep 17 14:35:40.126840 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:40.126806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:40.135488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:40.135458    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"
Sep 17 14:35:40.334004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:40.333961    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy" containerID="cri-o://d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d" gracePeriod=30
Sep 17 14:35:40.437048 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:40.437020    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"]
Sep 17 14:35:40.438802 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:35:40.438776    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86598416_2866_4dd3_b9a4_28525899abda.slice/crio-7bf42c6b5a0ca5730a516a32b49036d198e41993f740c7d0ec4470993cca86d7 WatchSource:0}: Error finding container 7bf42c6b5a0ca5730a516a32b49036d198e41993f740c7d0ec4470993cca86d7: Status 404 returned error can't find the container with id 7bf42c6b5a0ca5730a516a32b49036d198e41993f740c7d0ec4470993cca86d7
Sep 17 14:35:40.927056 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:40.927019    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:41.342583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:41.342548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b" event={"ID":"86598416-2866-4dd3-b9a4-28525899abda","Type":"ContainerStarted","Data":"36574c73ebedc5f10a7fb47298a28c6572c3f51e5cccf788579f700b7172344d"}
Sep 17 14:35:41.342757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:41.342640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b" event={"ID":"86598416-2866-4dd3-b9a4-28525899abda","Type":"ContainerStarted","Data":"7bf42c6b5a0ca5730a516a32b49036d198e41993f740c7d0ec4470993cca86d7"}
Sep 17 14:35:41.623220 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:41.623127    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:42.335209 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:42.335175    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:42.351096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:42.350471    2565 generic.go:356] "Generic (PLEG): container finished" podID="86598416-2866-4dd3-b9a4-28525899abda" containerID="36574c73ebedc5f10a7fb47298a28c6572c3f51e5cccf788579f700b7172344d" exitCode=0
Sep 17 14:35:42.351096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:42.350691    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b" event={"ID":"86598416-2866-4dd3-b9a4-28525899abda","Type":"ContainerDied","Data":"36574c73ebedc5f10a7fb47298a28c6572c3f51e5cccf788579f700b7172344d"}
Sep 17 14:35:42.879599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:42.879555    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:42.977819 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:42.977773    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:43.078019 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:43.077978    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:43.511949 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:43.511923    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"
Sep 17 14:35:43.556863 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:43.556821    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86598416-2866-4dd3-b9a4-28525899abda-maas-api-service-ca\") pod \"86598416-2866-4dd3-b9a4-28525899abda\" (UID: \"86598416-2866-4dd3-b9a4-28525899abda\") "
Sep 17 14:35:43.557225 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:43.557200    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86598416-2866-4dd3-b9a4-28525899abda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "86598416-2866-4dd3-b9a4-28525899abda" (UID: "86598416-2866-4dd3-b9a4-28525899abda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:43.658348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:43.658242    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/86598416-2866-4dd3-b9a4-28525899abda-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:43.739024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:43.738991    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:44.365884 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:44.365851    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"
Sep 17 14:35:44.366322 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:44.365853    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b" event={"ID":"86598416-2866-4dd3-b9a4-28525899abda","Type":"ContainerDied","Data":"7bf42c6b5a0ca5730a516a32b49036d198e41993f740c7d0ec4470993cca86d7"}
Sep 17 14:35:44.366322 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:44.365974    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bf42c6b5a0ca5730a516a32b49036d198e41993f740c7d0ec4470993cca86d7"
Sep 17 14:35:44.703287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:44.703157    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:45.395428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:45.395396    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:48.597643 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.597577    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:35:48.714021 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.713981    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-socket\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714081    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-certs\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714113    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-token\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istiod-ca-cert\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-data\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714200    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-podinfo\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-credential-socket\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714352    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:35:48.714451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fsbx\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-kube-api-access-9fsbx\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-envoy\") pod \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\" (UID: \"d9e2acd3-48ba-462f-a17a-7c0ca48427ed\") "
Sep 17 14:35:48.714709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714486    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:35:48.714709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714501    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:35:48.714709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:35:48.714709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714686    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:35:48.714957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714938    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:48.715024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714963    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:48.715024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714977    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:48.715024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.714989    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:48.715024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.715004    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:48.716539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.716513    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:48.716679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.716511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:35:48.716961 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.716943    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:35:48.716961 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.716947    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-kube-api-access-9fsbx" (OuterVolumeSpecName: "kube-api-access-9fsbx") pod "d9e2acd3-48ba-462f-a17a-7c0ca48427ed" (UID: "d9e2acd3-48ba-462f-a17a-7c0ca48427ed"). InnerVolumeSpecName "kube-api-access-9fsbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:35:48.816494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.816455    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:48.816494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.816489    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fsbx\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-kube-api-access-9fsbx\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:48.816494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.816500    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:48.816757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:48.816509    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2acd3-48ba-462f-a17a-7c0ca48427ed-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:35:49.402010 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.401976    2565 generic.go:356] "Generic (PLEG): container finished" podID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerID="d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d" exitCode=0
Sep 17 14:35:49.402202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.402028    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"
Sep 17 14:35:49.402202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.402065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" event={"ID":"d9e2acd3-48ba-462f-a17a-7c0ca48427ed","Type":"ContainerDied","Data":"d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d"}
Sep 17 14:35:49.402202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.402120    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz" event={"ID":"d9e2acd3-48ba-462f-a17a-7c0ca48427ed","Type":"ContainerDied","Data":"0ec5e964e62e385913da9afdeba47bc31633bb2da1184ca4e1b050da09233b85"}
Sep 17 14:35:49.402202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.402144    2565 scope.go:117] "RemoveContainer" containerID="d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d"
Sep 17 14:35:49.413171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.413153    2565 scope.go:117] "RemoveContainer" containerID="03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35"
Sep 17 14:35:49.422194 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.422173    2565 scope.go:117] "RemoveContainer" containerID="d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d"
Sep 17 14:35:49.423118 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:35:49.422628    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d\": container with ID starting with d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d not found: ID does not exist" containerID="d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d"
Sep 17 14:35:49.423118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.422697    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d"} err="failed to get container status \"d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d\": rpc error: code = NotFound desc = could not find container \"d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d\": container with ID starting with d887fbaf46538b3700614caf114dd0a7745ff564b06f2d784946eefa073d2b2d not found: ID does not exist"
Sep 17 14:35:49.423118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.422727    2565 scope.go:117] "RemoveContainer" containerID="03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35"
Sep 17 14:35:49.423118 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:35:49.423051    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35\": container with ID starting with 03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35 not found: ID does not exist" containerID="03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35"
Sep 17 14:35:49.423118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.423084    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35"} err="failed to get container status \"03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35\": rpc error: code = NotFound desc = could not find container \"03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35\": container with ID starting with 03af3b10ea0c8b19f5cbfaebb13c171f479774e747955a609434fbdb72e13f35 not found: ID does not exist"
Sep 17 14:35:49.425737 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.425717    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"]
Sep 17 14:35:49.431642 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.431615    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0829c1-openshift-default-7bd67bbb64-2vlgz"]
Sep 17 14:35:49.520734 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:49.520697    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" path="/var/lib/kubelet/pods/d9e2acd3-48ba-462f-a17a-7c0ca48427ed/volumes"
Sep 17 14:35:50.061124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:50.061091    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:50.652714 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:50.652670    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:50.720686 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:50.720653    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:50.788524 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:50.788491    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:50.854979 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:50.854946    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"]
Sep 17 14:35:50.857081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:50.857049    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:50.857588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:50.857567    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4009da-trg7b"]
Sep 17 14:35:50.926514 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:50.926433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:51.219422 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:51.219325    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:51.357926 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:51.357892    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:51.520089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:51.520009    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86598416-2866-4dd3-b9a4-28525899abda" path="/var/lib/kubelet/pods/86598416-2866-4dd3-b9a4-28525899abda/volumes"
Sep 17 14:35:52.141324 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:52.141282    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:52.920551 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:52.920492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:53.564125 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:53.564090    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:53.921091 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:53.921055    2565 scope.go:117] "RemoveContainer" containerID="d8435814e8889d18809fef93d77d4cc9235d820798230d3c8cdd23bb6e4265c5"
Sep 17 14:35:54.334947 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:54.334905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:57.018738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.018707    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:57.084817 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.084784    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:57.252910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.248559    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"]
Sep 17 14:35:57.263294 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.249235    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy" containerID="cri-o://af6c759015edeb655a9de4a9e7f5c7b0a41b6aa96ced5a54d61331aca08c9e86" gracePeriod=30
Sep 17 14:35:57.271144 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.271064    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:57.607340 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607305    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"]
Sep 17 14:35:57.607783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607761    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86598416-2866-4dd3-b9a4-28525899abda" containerName="revoke-keys"
Sep 17 14:35:57.607783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607780    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="86598416-2866-4dd3-b9a4-28525899abda" containerName="revoke-keys"
Sep 17 14:35:57.607997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607799    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy"
Sep 17 14:35:57.607997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607805    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy"
Sep 17 14:35:57.607997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607818    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy"
Sep 17 14:35:57.607997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607824    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy"
Sep 17 14:35:57.607997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607907    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy"
Sep 17 14:35:57.607997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607916    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="86598416-2866-4dd3-b9a4-28525899abda" containerName="revoke-keys"
Sep 17 14:35:57.607997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.607922    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e2acd3-48ba-462f-a17a-7c0ca48427ed" containerName="istio-proxy"
Sep 17 14:35:57.611201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.611181    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"
Sep 17 14:35:57.613315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.613294    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:35:57.613315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.613306    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:35:57.618571 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.618543    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"]
Sep 17 14:35:57.712740 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.712701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42eca6f-8c9d-42d1-a35e-49d2a5493bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc\" (UID: \"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"
Sep 17 14:35:57.813754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.813708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42eca6f-8c9d-42d1-a35e-49d2a5493bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc\" (UID: \"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"
Sep 17 14:35:57.814416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.814391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42eca6f-8c9d-42d1-a35e-49d2a5493bd2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc\" (UID: \"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"
Sep 17 14:35:57.896736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.896624    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:57.923664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:57.923614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"
Sep 17 14:35:58.205483 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:58.205457    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"]
Sep 17 14:35:58.207083 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:35:58.207055    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf42eca6f_8c9d_42d1_a35e_49d2a5493bd2.slice/crio-fc43752dc9cc61da355ddfb970c1998798c0845b30b5e074931d044de7672562 WatchSource:0}: Error finding container fc43752dc9cc61da355ddfb970c1998798c0845b30b5e074931d044de7672562: Status 404 returned error can't find the container with id fc43752dc9cc61da355ddfb970c1998798c0845b30b5e074931d044de7672562
Sep 17 14:35:58.463676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:58.463577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc" event={"ID":"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2","Type":"ContainerStarted","Data":"cdf1dbba26b8cb27f94f977a43737f2d45fa58594433ab3b2b072348c2bff65b"}
Sep 17 14:35:58.463676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:58.463638    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc" event={"ID":"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2","Type":"ContainerStarted","Data":"fc43752dc9cc61da355ddfb970c1998798c0845b30b5e074931d044de7672562"}
Sep 17 14:35:58.481248 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:58.481198    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc" podStartSLOduration=1.4811826240000001 podStartE2EDuration="1.481182624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:35:58.479689093 +0000 UTC m=+1745.413194286" watchObservedRunningTime="2026-09-17 14:35:58.481182624 +0000 UTC m=+1745.414687817"
Sep 17 14:35:58.731548 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:58.731450    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:35:59.471660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:59.471622    2565 generic.go:356] "Generic (PLEG): container finished" podID="f42eca6f-8c9d-42d1-a35e-49d2a5493bd2" containerID="cdf1dbba26b8cb27f94f977a43737f2d45fa58594433ab3b2b072348c2bff65b" exitCode=0
Sep 17 14:35:59.472289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:59.471665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc" event={"ID":"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2","Type":"ContainerDied","Data":"cdf1dbba26b8cb27f94f977a43737f2d45fa58594433ab3b2b072348c2bff65b"}
Sep 17 14:35:59.611219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:35:59.611185    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:00.433690 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:00.433654    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:00.622369 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:00.622343    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"
Sep 17 14:36:00.744881 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:00.744786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42eca6f-8c9d-42d1-a35e-49d2a5493bd2-maas-api-service-ca\") pod \"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2\" (UID: \"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2\") "
Sep 17 14:36:00.745189 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:00.745165    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f42eca6f-8c9d-42d1-a35e-49d2a5493bd2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f42eca6f-8c9d-42d1-a35e-49d2a5493bd2" (UID: "f42eca6f-8c9d-42d1-a35e-49d2a5493bd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:00.846520 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:00.846480    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f42eca6f-8c9d-42d1-a35e-49d2a5493bd2-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:01.239855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:01.239819    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:01.485455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:01.485411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc" event={"ID":"f42eca6f-8c9d-42d1-a35e-49d2a5493bd2","Type":"ContainerDied","Data":"fc43752dc9cc61da355ddfb970c1998798c0845b30b5e074931d044de7672562"}
Sep 17 14:36:01.485631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:01.485466    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc43752dc9cc61da355ddfb970c1998798c0845b30b5e074931d044de7672562"
Sep 17 14:36:01.485631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:01.485426    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"
Sep 17 14:36:01.945122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:01.945090    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:02.493961 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.493928    2565 generic.go:356] "Generic (PLEG): container finished" podID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerID="af6c759015edeb655a9de4a9e7f5c7b0a41b6aa96ced5a54d61331aca08c9e86" exitCode=0
Sep 17 14:36:02.494111 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.494005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" event={"ID":"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3","Type":"ContainerDied","Data":"af6c759015edeb655a9de4a9e7f5c7b0a41b6aa96ced5a54d61331aca08c9e86"}
Sep 17 14:36:02.494187 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.494114    2565 scope.go:117] "RemoveContainer" containerID="25ec8c694bc1aebf5b7275314880065b56b433a76d758bf7136042e28f5db628"
Sep 17 14:36:02.518846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.518826    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:36:02.667421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667383    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-credential-socket\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.667631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667435    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-data\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.667631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667470    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqq8w\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-kube-api-access-fqq8w\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.667631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667506    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-envoy\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.667631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667540    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-certs\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.667631 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667587    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-podinfo\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.667915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667671    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-token\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.667915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667713    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-socket\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.667915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667727    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:02.667915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667743    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:02.667915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667766    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istiod-ca-cert\") pod \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\" (UID: \"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3\") "
Sep 17 14:36:02.668176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.667910    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:02.668343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.668305    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:02.668433 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.668414    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:02.668489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.668419    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:02.668489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.668455    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:02.668489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.668472    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:02.669984 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.669954    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-kube-api-access-fqq8w" (OuterVolumeSpecName: "kube-api-access-fqq8w") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "kube-api-access-fqq8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:02.670282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.670231    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:02.670453 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.670439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:02.670641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.670624    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" (UID: "d4d6e611-9d6f-49dc-82b5-2a3ded3781d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:36:02.770083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.770037    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:02.770083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.770072    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqq8w\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-kube-api-access-fqq8w\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:02.770083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.770084    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:02.770083 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.770093    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:02.770533 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.770100    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:02.770533 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:02.770109    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:03.501029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:03.500997    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"
Sep 17 14:36:03.501477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:03.501034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2" event={"ID":"d4d6e611-9d6f-49dc-82b5-2a3ded3781d3","Type":"ContainerDied","Data":"ce1e38b00f1e4d392115c289922d1cd76618090b1a73c05d9471aae6852295f3"}
Sep 17 14:36:03.501477 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:03.501095    2565 scope.go:117] "RemoveContainer" containerID="af6c759015edeb655a9de4a9e7f5c7b0a41b6aa96ced5a54d61331aca08c9e86"
Sep 17 14:36:03.537227 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:03.537194    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"]
Sep 17 14:36:03.541456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:03.541426    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4009da-openshift-default-76fc699c86-sl9b2"]
Sep 17 14:36:05.520046 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:05.520013    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" path="/var/lib/kubelet/pods/d4d6e611-9d6f-49dc-82b5-2a3ded3781d3/volumes"
Sep 17 14:36:08.098513 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.098479    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"]
Sep 17 14:36:08.099940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.099902    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:08.100853 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.100831    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-60bf73-29827590-8qv5p"]
Sep 17 14:36:08.125707 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.125666    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"]
Sep 17 14:36:08.126039 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.125979    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv" podUID="366a6979-6413-4ade-831b-584ffcd9be6e" containerName="payload-processing" containerID="cri-o://ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04" gracePeriod=30
Sep 17 14:36:08.133821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.133357    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"]
Sep 17 14:36:08.133821 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.133585    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq" podUID="97a45b2f-ed71-4182-b317-9527280f9ab5" containerName="payload-pre-processing" containerID="cri-o://91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f" gracePeriod=30
Sep 17 14:36:08.174000 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.173972    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:08.247949 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.247918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:08.290600 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.290565    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"]
Sep 17 14:36:08.295173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.295147    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-60bf73-s57gc"]
Sep 17 14:36:08.316597 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.316502    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:08.424209 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.424181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:36:08.427390 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.427358    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:36:08.532582 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.532547    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366a6979-6413-4ade-831b-584ffcd9be6e-plugins-config-volume\") pod \"366a6979-6413-4ade-831b-584ffcd9be6e\" (UID: \"366a6979-6413-4ade-831b-584ffcd9be6e\") "
Sep 17 14:36:08.532811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.532657    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a45b2f-ed71-4182-b317-9527280f9ab5-plugins-config-volume\") pod \"97a45b2f-ed71-4182-b317-9527280f9ab5\" (UID: \"97a45b2f-ed71-4182-b317-9527280f9ab5\") "
Sep 17 14:36:08.532811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.532714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kw2ph\" (UniqueName: \"kubernetes.io/projected/366a6979-6413-4ade-831b-584ffcd9be6e-kube-api-access-kw2ph\") pod \"366a6979-6413-4ade-831b-584ffcd9be6e\" (UID: \"366a6979-6413-4ade-831b-584ffcd9be6e\") "
Sep 17 14:36:08.532962 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.532814    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kz6l6\" (UniqueName: \"kubernetes.io/projected/97a45b2f-ed71-4182-b317-9527280f9ab5-kube-api-access-kz6l6\") pod \"97a45b2f-ed71-4182-b317-9527280f9ab5\" (UID: \"97a45b2f-ed71-4182-b317-9527280f9ab5\") "
Sep 17 14:36:08.533059 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.533038    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97a45b2f-ed71-4182-b317-9527280f9ab5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "97a45b2f-ed71-4182-b317-9527280f9ab5" (UID: "97a45b2f-ed71-4182-b317-9527280f9ab5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:08.533121 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.533030    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/366a6979-6413-4ade-831b-584ffcd9be6e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "366a6979-6413-4ade-831b-584ffcd9be6e" (UID: "366a6979-6413-4ade-831b-584ffcd9be6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:08.533302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.533280    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97a45b2f-ed71-4182-b317-9527280f9ab5-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:08.533302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.533304    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/366a6979-6413-4ade-831b-584ffcd9be6e-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:08.535567 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.535536    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/366a6979-6413-4ade-831b-584ffcd9be6e-kube-api-access-kw2ph" (OuterVolumeSpecName: "kube-api-access-kw2ph") pod "366a6979-6413-4ade-831b-584ffcd9be6e" (UID: "366a6979-6413-4ade-831b-584ffcd9be6e"). InnerVolumeSpecName "kube-api-access-kw2ph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:08.535739 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.535573    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97a45b2f-ed71-4182-b317-9527280f9ab5-kube-api-access-kz6l6" (OuterVolumeSpecName: "kube-api-access-kz6l6") pod "97a45b2f-ed71-4182-b317-9527280f9ab5" (UID: "97a45b2f-ed71-4182-b317-9527280f9ab5"). InnerVolumeSpecName "kube-api-access-kz6l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:08.536458 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.536424    2565 generic.go:356] "Generic (PLEG): container finished" podID="366a6979-6413-4ade-831b-584ffcd9be6e" containerID="ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04" exitCode=0
Sep 17 14:36:08.536566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.536485    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"
Sep 17 14:36:08.536566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.536510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv" event={"ID":"366a6979-6413-4ade-831b-584ffcd9be6e","Type":"ContainerDied","Data":"ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04"}
Sep 17 14:36:08.536693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.536567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv" event={"ID":"366a6979-6413-4ade-831b-584ffcd9be6e","Type":"ContainerDied","Data":"3f24c2fd0ba8449c6dab62d743320350d8d002b091d5024ddf6a15f587c5fc10"}
Sep 17 14:36:08.536693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.536586    2565 scope.go:117] "RemoveContainer" containerID="ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04"
Sep 17 14:36:08.538328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.538303    2565 generic.go:356] "Generic (PLEG): container finished" podID="97a45b2f-ed71-4182-b317-9527280f9ab5" containerID="91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f" exitCode=0
Sep 17 14:36:08.538452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.538349    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"
Sep 17 14:36:08.538452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.538364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq" event={"ID":"97a45b2f-ed71-4182-b317-9527280f9ab5","Type":"ContainerDied","Data":"91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f"}
Sep 17 14:36:08.538452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.538433    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq" event={"ID":"97a45b2f-ed71-4182-b317-9527280f9ab5","Type":"ContainerDied","Data":"b28c47b5dbde324bcf1af9ae894d004d75d82113aff1ed3055d2821f7fa56891"}
Sep 17 14:36:08.546932 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.546907    2565 scope.go:117] "RemoveContainer" containerID="ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04"
Sep 17 14:36:08.547382 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:36:08.547356    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04\": container with ID starting with ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04 not found: ID does not exist" containerID="ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04"
Sep 17 14:36:08.547495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.547388    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04"} err="failed to get container status \"ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04\": rpc error: code = NotFound desc = could not find container \"ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04\": container with ID starting with ba3d8ca1994338dae4a2e7826c0e97c9c6a911758871ba421e475e5a88e89b04 not found: ID does not exist"
Sep 17 14:36:08.547495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.547406    2565 scope.go:117] "RemoveContainer" containerID="91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f"
Sep 17 14:36:08.561499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.561467    2565 scope.go:117] "RemoveContainer" containerID="91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f"
Sep 17 14:36:08.562237 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:36:08.562150    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f\": container with ID starting with 91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f not found: ID does not exist" containerID="91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f"
Sep 17 14:36:08.562237 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.562201    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f"} err="failed to get container status \"91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f\": rpc error: code = NotFound desc = could not find container \"91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f\": container with ID starting with 91789c242982eb36f8a78269756eeee3954bfe50fe898c37748e7c90c0d94a5f not found: ID does not exist"
Sep 17 14:36:08.566644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.566617    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"]
Sep 17 14:36:08.572889 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.572856    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-60bf73-ddfdf7bbc-rsrsv"]
Sep 17 14:36:08.581612 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.581585    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"]
Sep 17 14:36:08.585742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.585716    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-60bf73-6b4f8f6ff4-jxmqq"]
Sep 17 14:36:08.635098 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.635053    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw2ph\" (UniqueName: \"kubernetes.io/projected/366a6979-6413-4ade-831b-584ffcd9be6e-kube-api-access-kw2ph\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:08.635098 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.635086    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kz6l6\" (UniqueName: \"kubernetes.io/projected/97a45b2f-ed71-4182-b317-9527280f9ab5-kube-api-access-kz6l6\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:08.644955 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.644857    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:08.745468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:08.745433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:09.489688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:09.489597    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:09.520896 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:09.520867    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="366a6979-6413-4ade-831b-584ffcd9be6e" path="/var/lib/kubelet/pods/366a6979-6413-4ade-831b-584ffcd9be6e/volumes"
Sep 17 14:36:09.521193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:09.521181    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e22d522-f668-4515-81c9-6596acbcf390" path="/var/lib/kubelet/pods/4e22d522-f668-4515-81c9-6596acbcf390/volumes"
Sep 17 14:36:09.521545 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:09.521530    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97a45b2f-ed71-4182-b317-9527280f9ab5" path="/var/lib/kubelet/pods/97a45b2f-ed71-4182-b317-9527280f9ab5/volumes"
Sep 17 14:36:09.521888 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:09.521875    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f42eca6f-8c9d-42d1-a35e-49d2a5493bd2" path="/var/lib/kubelet/pods/f42eca6f-8c9d-42d1-a35e-49d2a5493bd2/volumes"
Sep 17 14:36:10.138209 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:10.138172    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:10.935355 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:10.935326    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:14.993196 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:14.993159    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:15.095115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:15.095078    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:15.236871 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:15.236836    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"]
Sep 17 14:36:15.237128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:15.237102    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy" containerID="cri-o://d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c" gracePeriod=30
Sep 17 14:36:15.263764 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:15.263672    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:20.490790 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.490761    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:36:20.566489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566423    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-envoy\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566457    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-istio-token\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566543    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-certs\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566562    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swm2j\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-kube-api-access-swm2j\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566586    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71952167-bbbe-4528-a9e1-5c79de58326b-istiod-ca-cert\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566610    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71952167-bbbe-4528-a9e1-5c79de58326b-istio-podinfo\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566639    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-socket\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566658    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-data\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566723    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-credential-socket\") pod \"71952167-bbbe-4528-a9e1-5c79de58326b\" (UID: \"71952167-bbbe-4528-a9e1-5c79de58326b\") "
Sep 17 14:36:20.566990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566779    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:20.567094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.566994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:20.567094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.567011    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-data" (OuterVolumeSpecName: "istio-data") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:20.567199 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.567157    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71952167-bbbe-4528-a9e1-5c79de58326b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:20.567285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.567203    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:20.567285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.567226    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:20.567285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.567248    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:20.567464 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.567294    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:20.568760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.568735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-kube-api-access-swm2j" (OuterVolumeSpecName: "kube-api-access-swm2j") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "kube-api-access-swm2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:20.568760 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.568739    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71952167-bbbe-4528-a9e1-5c79de58326b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:36:20.568949 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.568928    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:20.569052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.569032    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-istio-token" (OuterVolumeSpecName: "istio-token") pod "71952167-bbbe-4528-a9e1-5c79de58326b" (UID: "71952167-bbbe-4528-a9e1-5c79de58326b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:20.617801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.617779    2565 generic.go:356] "Generic (PLEG): container finished" podID="71952167-bbbe-4528-a9e1-5c79de58326b" containerID="d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c" exitCode=0
Sep 17 14:36:20.617907 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.617826    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"
Sep 17 14:36:20.617907 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.617856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" event={"ID":"71952167-bbbe-4528-a9e1-5c79de58326b","Type":"ContainerDied","Data":"d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c"}
Sep 17 14:36:20.617907 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.617897    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc" event={"ID":"71952167-bbbe-4528-a9e1-5c79de58326b","Type":"ContainerDied","Data":"2c5af91cd9e4959c36ef6e660ea6cae18389f10d7d5247630ade9832265bb3f0"}
Sep 17 14:36:20.618019 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.617918    2565 scope.go:117] "RemoveContainer" containerID="d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c"
Sep 17 14:36:20.627090 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.627074    2565 scope.go:117] "RemoveContainer" containerID="8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1"
Sep 17 14:36:20.635458 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.635442    2565 scope.go:117] "RemoveContainer" containerID="d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c"
Sep 17 14:36:20.635707 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:36:20.635691    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c\": container with ID starting with d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c not found: ID does not exist" containerID="d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c"
Sep 17 14:36:20.635757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.635714    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c"} err="failed to get container status \"d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c\": rpc error: code = NotFound desc = could not find container \"d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c\": container with ID starting with d92ea358b53f9c8658f4b75378fba93125f4f755c17be0a0eb0ef1ee28130c7c not found: ID does not exist"
Sep 17 14:36:20.635757 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.635729    2565 scope.go:117] "RemoveContainer" containerID="8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1"
Sep 17 14:36:20.635981 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:36:20.635963    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1\": container with ID starting with 8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1 not found: ID does not exist" containerID="8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1"
Sep 17 14:36:20.636029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.635984    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1"} err="failed to get container status \"8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1\": rpc error: code = NotFound desc = could not find container \"8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1\": container with ID starting with 8c0e15a0ca5ecbd6e7a1374f64e0cf12d65c7c12387459e52212a30351fba1f1 not found: ID does not exist"
Sep 17 14:36:20.640454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.640434    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"]
Sep 17 14:36:20.647117 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.647095    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-60bf73-openshift-default-66966687fb-zz7cc"]
Sep 17 14:36:20.668171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.668148    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swm2j\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-kube-api-access-swm2j\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:20.668171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.668168    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71952167-bbbe-4528-a9e1-5c79de58326b-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:20.668300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.668179    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71952167-bbbe-4528-a9e1-5c79de58326b-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:20.668300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.668187    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:20.668300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.668196    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71952167-bbbe-4528-a9e1-5c79de58326b-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:20.668300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:20.668203    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71952167-bbbe-4528-a9e1-5c79de58326b-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:21.520041 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:21.520008    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" path="/var/lib/kubelet/pods/71952167-bbbe-4528-a9e1-5c79de58326b/volumes"
Sep 17 14:36:26.820620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.820587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:26.904546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904505    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"]
Sep 17 14:36:26.904928 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904908    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy"
Sep 17 14:36:26.904928 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904924    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904935    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="366a6979-6413-4ade-831b-584ffcd9be6e" containerName="payload-processing"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904941    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="366a6979-6413-4ade-831b-584ffcd9be6e" containerName="payload-processing"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904956    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904962    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904975    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f42eca6f-8c9d-42d1-a35e-49d2a5493bd2" containerName="revoke-keys"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904981    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f42eca6f-8c9d-42d1-a35e-49d2a5493bd2" containerName="revoke-keys"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904991    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.904996    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905005    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97a45b2f-ed71-4182-b317-9527280f9ab5" containerName="payload-pre-processing"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905011    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97a45b2f-ed71-4182-b317-9527280f9ab5" containerName="payload-pre-processing"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905016    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy"
Sep 17 14:36:26.905029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905022    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy"
Sep 17 14:36:26.905430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905082    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97a45b2f-ed71-4182-b317-9527280f9ab5" containerName="payload-pre-processing"
Sep 17 14:36:26.905430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905092    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy"
Sep 17 14:36:26.905430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905099    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d6e611-9d6f-49dc-82b5-2a3ded3781d3" containerName="istio-proxy"
Sep 17 14:36:26.905430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905106    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy"
Sep 17 14:36:26.905430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905112    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="366a6979-6413-4ade-831b-584ffcd9be6e" containerName="payload-processing"
Sep 17 14:36:26.905430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905118    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f42eca6f-8c9d-42d1-a35e-49d2a5493bd2" containerName="revoke-keys"
Sep 17 14:36:26.905430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.905123    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71952167-bbbe-4528-a9e1-5c79de58326b" containerName="istio-proxy"
Sep 17 14:36:26.908139 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.908123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"
Sep 17 14:36:26.910142 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.910112    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:36:26.910274 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.910114    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:36:26.914964 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:26.914942    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"]
Sep 17 14:36:27.039298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:27.039246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88844dda-dea8-4f9f-9fa2-8aac82f87cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q\" (UID: \"88844dda-dea8-4f9f-9fa2-8aac82f87cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"
Sep 17 14:36:27.140797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:27.140754    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88844dda-dea8-4f9f-9fa2-8aac82f87cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q\" (UID: \"88844dda-dea8-4f9f-9fa2-8aac82f87cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"
Sep 17 14:36:27.141426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:27.141404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88844dda-dea8-4f9f-9fa2-8aac82f87cf2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q\" (UID: \"88844dda-dea8-4f9f-9fa2-8aac82f87cf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"
Sep 17 14:36:27.211387 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:27.211352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:27.221240 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:27.221209    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"
Sep 17 14:36:27.494742 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:27.494716    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"]
Sep 17 14:36:27.496644 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:36:27.496617    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88844dda_dea8_4f9f_9fa2_8aac82f87cf2.slice/crio-7fd2e9b974e860f896202edc7c01036b4cf129f23a9f43c17668598b162a24eb WatchSource:0}: Error finding container 7fd2e9b974e860f896202edc7c01036b4cf129f23a9f43c17668598b162a24eb: Status 404 returned error can't find the container with id 7fd2e9b974e860f896202edc7c01036b4cf129f23a9f43c17668598b162a24eb
Sep 17 14:36:27.660266 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:27.660220    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q" event={"ID":"88844dda-dea8-4f9f-9fa2-8aac82f87cf2","Type":"ContainerStarted","Data":"7fd2e9b974e860f896202edc7c01036b4cf129f23a9f43c17668598b162a24eb"}
Sep 17 14:36:28.027001 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:28.026884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:28.667180 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:28.667147    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:28.668060 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:28.668033    2565 generic.go:356] "Generic (PLEG): container finished" podID="88844dda-dea8-4f9f-9fa2-8aac82f87cf2" containerID="f5f3056fdb8fbd33ecce54902b7eb79ad433856023ddf532a17733bf039eebe8" exitCode=0
Sep 17 14:36:28.668186 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:28.668131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q" event={"ID":"88844dda-dea8-4f9f-9fa2-8aac82f87cf2","Type":"ContainerDied","Data":"f5f3056fdb8fbd33ecce54902b7eb79ad433856023ddf532a17733bf039eebe8"}
Sep 17 14:36:29.366783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:29.366727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:29.809361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:29.809337    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"
Sep 17 14:36:29.868777 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:29.868749    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88844dda-dea8-4f9f-9fa2-8aac82f87cf2-maas-api-service-ca\") pod \"88844dda-dea8-4f9f-9fa2-8aac82f87cf2\" (UID: \"88844dda-dea8-4f9f-9fa2-8aac82f87cf2\") "
Sep 17 14:36:29.869120 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:29.869098    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88844dda-dea8-4f9f-9fa2-8aac82f87cf2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88844dda-dea8-4f9f-9fa2-8aac82f87cf2" (UID: "88844dda-dea8-4f9f-9fa2-8aac82f87cf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:29.970278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:29.970172    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88844dda-dea8-4f9f-9fa2-8aac82f87cf2-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:30.059864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:30.059833    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:30.687363 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:30.687279    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"
Sep 17 14:36:30.687363 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:30.687301    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q" event={"ID":"88844dda-dea8-4f9f-9fa2-8aac82f87cf2","Type":"ContainerDied","Data":"7fd2e9b974e860f896202edc7c01036b4cf129f23a9f43c17668598b162a24eb"}
Sep 17 14:36:30.687787 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:30.687370    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fd2e9b974e860f896202edc7c01036b4cf129f23a9f43c17668598b162a24eb"
Sep 17 14:36:31.774910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:31.774871    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"]
Sep 17 14:36:31.801834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:31.801797    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"]
Sep 17 14:36:31.850918 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:31.850884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:42.475803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:42.475758    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"]
Sep 17 14:36:42.477918 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:42.477892    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b50f2-29827590-7wqpw"]
Sep 17 14:36:42.481626 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:42.481594    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:42.543646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:42.543606    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:42.647157 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:42.647116    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:42.658466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:42.658419    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"]
Sep 17 14:36:42.660572 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:42.660546    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b50f2-4q22q"]
Sep 17 14:36:42.744438 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:42.744344    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:43.015376 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:43.015281    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:43.114463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:43.114395    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:43.521893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:43.521859    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63110548-178d-4f6d-a2a5-68f3a4aa7f8c" path="/var/lib/kubelet/pods/63110548-178d-4f6d-a2a5-68f3a4aa7f8c/volumes"
Sep 17 14:36:43.522318 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:43.522187    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88844dda-dea8-4f9f-9fa2-8aac82f87cf2" path="/var/lib/kubelet/pods/88844dda-dea8-4f9f-9fa2-8aac82f87cf2/volumes"
Sep 17 14:36:43.928815 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:43.928776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:44.661109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:44.661078    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:46.794687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:46.794596    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr" podUID="e9f89105-a7b9-4f2b-b445-0dc45f5a06de" containerName="main" containerID="cri-o://dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7" gracePeriod=45
Sep 17 14:36:46.820921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:46.820887    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr" podUID="d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" containerName="main" containerID="cri-o://7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c" gracePeriod=45
Sep 17 14:36:47.069684 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.069655    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:36:47.100679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.100647    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:36:47.145777 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.145746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-dshm\") pod \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") "
Sep 17 14:36:47.145933 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.145786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-model-cache\") pod \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") "
Sep 17 14:36:47.145933 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.145909    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tmp-dir\") pod \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") "
Sep 17 14:36:47.146075 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.145957    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-home\") pod \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") "
Sep 17 14:36:47.146075 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.145996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gw7gx\" (UniqueName: \"kubernetes.io/projected/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-kube-api-access-gw7gx\") pod \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") "
Sep 17 14:36:47.146075 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.146045    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tls-certs\") pod \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\" (UID: \"e9f89105-a7b9-4f2b-b445-0dc45f5a06de\") "
Sep 17 14:36:47.146292 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.146227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9f89105-a7b9-4f2b-b445-0dc45f5a06de" (UID: "e9f89105-a7b9-4f2b-b445-0dc45f5a06de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.146577 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.146540    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-home" (OuterVolumeSpecName: "home") pod "e9f89105-a7b9-4f2b-b445-0dc45f5a06de" (UID: "e9f89105-a7b9-4f2b-b445-0dc45f5a06de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.146702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.146653    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9f89105-a7b9-4f2b-b445-0dc45f5a06de" (UID: "e9f89105-a7b9-4f2b-b445-0dc45f5a06de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.148016 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.147989    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-dshm" (OuterVolumeSpecName: "dshm") pod "e9f89105-a7b9-4f2b-b445-0dc45f5a06de" (UID: "e9f89105-a7b9-4f2b-b445-0dc45f5a06de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.148709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.148682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9f89105-a7b9-4f2b-b445-0dc45f5a06de" (UID: "e9f89105-a7b9-4f2b-b445-0dc45f5a06de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:36:47.148865 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.148842    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-kube-api-access-gw7gx" (OuterVolumeSpecName: "kube-api-access-gw7gx") pod "e9f89105-a7b9-4f2b-b445-0dc45f5a06de" (UID: "e9f89105-a7b9-4f2b-b445-0dc45f5a06de"). InnerVolumeSpecName "kube-api-access-gw7gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:47.247452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247414    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-dshm\") pod \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") "
Sep 17 14:36:47.247452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-home\") pod \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") "
Sep 17 14:36:47.247687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247517    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzrhf\" (UniqueName: \"kubernetes.io/projected/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-kube-api-access-mzrhf\") pod \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") "
Sep 17 14:36:47.247687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247538    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tls-certs\") pod \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") "
Sep 17 14:36:47.247687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247566    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tmp-dir\") pod \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") "
Sep 17 14:36:47.247687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247590    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-model-cache\") pod \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\" (UID: \"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b\") "
Sep 17 14:36:47.247898 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247741    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-home" (OuterVolumeSpecName: "home") pod "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" (UID: "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.247898 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247806    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" (UID: "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.247898 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.247864    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-model-cache" (OuterVolumeSpecName: "model-cache") pod "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" (UID: "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.248059 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248028    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.248059 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248046    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.248155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248061    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.248155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248074    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw7gx\" (UniqueName: \"kubernetes.io/projected/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-kube-api-access-gw7gx\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.248155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248088    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.248155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248102    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.248155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248114    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.248155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248125    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.248155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.248137    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9f89105-a7b9-4f2b-b445-0dc45f5a06de-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.249545 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.249520    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-kube-api-access-mzrhf" (OuterVolumeSpecName: "kube-api-access-mzrhf") pod "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" (UID: "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b"). InnerVolumeSpecName "kube-api-access-mzrhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:47.249668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.249573    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" (UID: "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:36:47.249799 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.249780    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-dshm" (OuterVolumeSpecName: "dshm") pod "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" (UID: "d59b729d-1f73-4ac4-a9a8-9bc05f32e70b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:47.348857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.348826    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzrhf\" (UniqueName: \"kubernetes.io/projected/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-kube-api-access-mzrhf\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.348857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.348852    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.349070 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.348863    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:47.797471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.797389    2565 generic.go:356] "Generic (PLEG): container finished" podID="d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" containerID="7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c" exitCode=0
Sep 17 14:36:47.797471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.797434    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"
Sep 17 14:36:47.797963 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.797475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr" event={"ID":"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b","Type":"ContainerDied","Data":"7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c"}
Sep 17 14:36:47.797963 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.797520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr" event={"ID":"d59b729d-1f73-4ac4-a9a8-9bc05f32e70b","Type":"ContainerDied","Data":"6425f7adc075b39bc6842605920f685287be4c3b2e2f916b596d4c9ff84248c7"}
Sep 17 14:36:47.797963 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.797544    2565 scope.go:117] "RemoveContainer" containerID="7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c"
Sep 17 14:36:47.799004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.798983    2565 generic.go:356] "Generic (PLEG): container finished" podID="e9f89105-a7b9-4f2b-b445-0dc45f5a06de" containerID="dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7" exitCode=0
Sep 17 14:36:47.799086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.799028    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"
Sep 17 14:36:47.799086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.799065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr" event={"ID":"e9f89105-a7b9-4f2b-b445-0dc45f5a06de","Type":"ContainerDied","Data":"dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7"}
Sep 17 14:36:47.799176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.799098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr" event={"ID":"e9f89105-a7b9-4f2b-b445-0dc45f5a06de","Type":"ContainerDied","Data":"42864058c2805b9aa79fd397d3efd975f8f0c2be32a1cf0008990f8d396b910d"}
Sep 17 14:36:47.814797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.814772    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"]
Sep 17 14:36:47.817312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.817294    2565 scope.go:117] "RemoveContainer" containerID="7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c"
Sep 17 14:36:47.817617 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:36:47.817597    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c\": container with ID starting with 7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c not found: ID does not exist" containerID="7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c"
Sep 17 14:36:47.817667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.817628    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c"} err="failed to get container status \"7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c\": rpc error: code = NotFound desc = could not find container \"7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c\": container with ID starting with 7fd8a833cff2bd4f910306ed6eb4dc87c774eb6c9cd6b59f42d57353c7952e8c not found: ID does not exist"
Sep 17 14:36:47.817667 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.817646    2565 scope.go:117] "RemoveContainer" containerID="dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7"
Sep 17 14:36:47.820781 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.820759    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/simulator-w4-8b50f2-kserve-875d497b4-t94rr"]
Sep 17 14:36:47.826451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.826433    2565 scope.go:117] "RemoveContainer" containerID="dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7"
Sep 17 14:36:47.826710 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:36:47.826691    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7\": container with ID starting with dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7 not found: ID does not exist" containerID="dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7"
Sep 17 14:36:47.826774 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.826720    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7"} err="failed to get container status \"dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7\": rpc error: code = NotFound desc = could not find container \"dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7\": container with ID starting with dd1256e72fdced983590f8ffb2b5b935603e80ae8a3af21cd03f2d20b1b123e7 not found: ID does not exist"
Sep 17 14:36:47.830372 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.830350    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"]
Sep 17 14:36:47.834599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:47.834577    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b50f2/distinct-2-w4-8b50f2-kserve-cf67444f9-zzbkr"]
Sep 17 14:36:49.216006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:49.215967    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:49.277169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:49.277139    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:49.463231 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:49.463198    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"]
Sep 17 14:36:49.463532 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:49.463507    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" containerID="cri-o://eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d" gracePeriod=30
Sep 17 14:36:49.473903 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:49.473828    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:36:49.520369 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:49.520339    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" path="/var/lib/kubelet/pods/d59b729d-1f73-4ac4-a9a8-9bc05f32e70b/volumes"
Sep 17 14:36:49.520783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:49.520768    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f89105-a7b9-4f2b-b445-0dc45f5a06de" path="/var/lib/kubelet/pods/e9f89105-a7b9-4f2b-b445-0dc45f5a06de/volumes"
Sep 17 14:36:53.624526 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:53.624493    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:36:53.629459 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:53.629438    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:36:53.630475 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:53.630453    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:36:53.639006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:53.638984    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:36:53.966654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:53.966558    2565 scope.go:117] "RemoveContainer" containerID="1b904704956ca85948330a5c8854ccd55f67d38caf2bf686c5d5a9e4ca045e8b"
Sep 17 14:36:53.976865 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:53.976829    2565 scope.go:117] "RemoveContainer" containerID="df131e6a30a2ce4f9ac14e0463c283504468e795d9934d8b77c79698b34efac4"
Sep 17 14:36:53.985504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:53.985476    2565 scope.go:117] "RemoveContainer" containerID="8a65805b853bbc176203676b2bb3eb24f3d49abf07742ce1b2f772804b6f801e"
Sep 17 14:36:54.383052 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.383016    2565 patch_prober.go:28] interesting pod/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:36:54.383232 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.383090    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:36:54.720356 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.720325    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:36:54.827352 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827326    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-podinfo\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827365    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-data\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827420    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istiod-ca-cert\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827535 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827519    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-certs\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827580    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-socket\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827629    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldrcj\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-kube-api-access-ldrcj\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827652    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-data" (OuterVolumeSpecName: "istio-data") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:54.827710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827676    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-envoy\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827703    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-credential-socket\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827716    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:54.827960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-token\") pod \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\" (UID: \"6301c8c7-7838-4cc3-b149-8e1c506e3da7\") "
Sep 17 14:36:54.827960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827833    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:54.827960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827849    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:36:54.828129 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.827970    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:54.828193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.828161    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:54.828193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.828181    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:54.828323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.828196    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:54.828323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.828211    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:54.828323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.828225    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:54.829659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.829624    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:36:54.829853 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.829835    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:36:54.829931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.829868    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-kube-api-access-ldrcj" (OuterVolumeSpecName: "kube-api-access-ldrcj") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "kube-api-access-ldrcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:54.829931 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.829917    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-token" (OuterVolumeSpecName: "istio-token") pod "6301c8c7-7838-4cc3-b149-8e1c506e3da7" (UID: "6301c8c7-7838-4cc3-b149-8e1c506e3da7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:36:54.844699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.844664    2565 generic.go:356] "Generic (PLEG): container finished" podID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerID="eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d" exitCode=0
Sep 17 14:36:54.844805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.844705    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" event={"ID":"6301c8c7-7838-4cc3-b149-8e1c506e3da7","Type":"ContainerDied","Data":"eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d"}
Sep 17 14:36:54.844805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.844728    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"
Sep 17 14:36:54.844805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.844760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb" event={"ID":"6301c8c7-7838-4cc3-b149-8e1c506e3da7","Type":"ContainerDied","Data":"21ba67538595b006560c63c1c389bba4aa3fbdcf1618302030ae87cf8e1eaae2"}
Sep 17 14:36:54.844805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.844788    2565 scope.go:117] "RemoveContainer" containerID="eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d"
Sep 17 14:36:54.855773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.855751    2565 scope.go:117] "RemoveContainer" containerID="1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f"
Sep 17 14:36:54.863802 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.863784    2565 scope.go:117] "RemoveContainer" containerID="eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d"
Sep 17 14:36:54.864076 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:36:54.864049    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d\": container with ID starting with eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d not found: ID does not exist" containerID="eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d"
Sep 17 14:36:54.864174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.864088    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d"} err="failed to get container status \"eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d\": rpc error: code = NotFound desc = could not find container \"eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d\": container with ID starting with eee3aac7fec7af117356f354f064fcfc2d455a87a6ce8d2fee1b769fe009b08d not found: ID does not exist"
Sep 17 14:36:54.864174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.864111    2565 scope.go:117] "RemoveContainer" containerID="1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f"
Sep 17 14:36:54.864437 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:36:54.864411    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f\": container with ID starting with 1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f not found: ID does not exist" containerID="1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f"
Sep 17 14:36:54.864530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.864447    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f"} err="failed to get container status \"1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f\": rpc error: code = NotFound desc = could not find container \"1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f\": container with ID starting with 1e247c9284e862c824df893765385580ed07a54bba153909d6b2d0dd50ce6a8f not found: ID does not exist"
Sep 17 14:36:54.867294 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.867272    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"]
Sep 17 14:36:54.871335 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.871314    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b50f2-openshift-default-869f86786-g5ldb"]
Sep 17 14:36:54.929373 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.929351    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldrcj\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-kube-api-access-ldrcj\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:54.929373 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.929373    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:54.929531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.929382    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:54.929531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:54.929391    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6301c8c7-7838-4cc3-b149-8e1c506e3da7-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:36:55.519978 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:36:55.519945    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" path="/var/lib/kubelet/pods/6301c8c7-7838-4cc3-b149-8e1c506e3da7/volumes"
Sep 17 14:37:08.303331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:08.303299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:08.711644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:08.711595    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:08.812466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:08.812429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:09.300502 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:09.300467    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:14.453842 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:14.453807    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"]
Sep 17 14:37:14.472822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:14.472785    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:15.397642 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:15.397609    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"]
Sep 17 14:37:15.418319 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:15.418292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:15.780357 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:15.780280    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:29.474501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.474443    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx" podUID="3ca3153e-7e2e-4c8b-9c10-058f04c2d359" containerName="main" containerID="cri-o://107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80" gracePeriod=45
Sep 17 14:37:29.735968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.735894    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:37:29.878511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878480    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vspfj\" (UniqueName: \"kubernetes.io/projected/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-kube-api-access-vspfj\") pod \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") "
Sep 17 14:37:29.878693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878550    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-home\") pod \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") "
Sep 17 14:37:29.878693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878591    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-dshm\") pod \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") "
Sep 17 14:37:29.878693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878610    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tls-certs\") pod \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") "
Sep 17 14:37:29.878868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878702    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-model-cache\") pod \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") "
Sep 17 14:37:29.878868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878735    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tmp-dir\") pod \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\" (UID: \"3ca3153e-7e2e-4c8b-9c10-058f04c2d359\") "
Sep 17 14:37:29.878985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878913    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-home" (OuterVolumeSpecName: "home") pod "3ca3153e-7e2e-4c8b-9c10-058f04c2d359" (UID: "3ca3153e-7e2e-4c8b-9c10-058f04c2d359"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:37:29.879042 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878971    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-model-cache" (OuterVolumeSpecName: "model-cache") pod "3ca3153e-7e2e-4c8b-9c10-058f04c2d359" (UID: "3ca3153e-7e2e-4c8b-9c10-058f04c2d359"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:37:29.879042 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.878994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3ca3153e-7e2e-4c8b-9c10-058f04c2d359" (UID: "3ca3153e-7e2e-4c8b-9c10-058f04c2d359"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:37:29.879145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.879099    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:29.879145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.879110    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:29.879145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.879118    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:29.880965 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.880935    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-dshm" (OuterVolumeSpecName: "dshm") pod "3ca3153e-7e2e-4c8b-9c10-058f04c2d359" (UID: "3ca3153e-7e2e-4c8b-9c10-058f04c2d359"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:37:29.881085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.880994    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3ca3153e-7e2e-4c8b-9c10-058f04c2d359" (UID: "3ca3153e-7e2e-4c8b-9c10-058f04c2d359"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:37:29.881085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.881004    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-kube-api-access-vspfj" (OuterVolumeSpecName: "kube-api-access-vspfj") pod "3ca3153e-7e2e-4c8b-9c10-058f04c2d359" (UID: "3ca3153e-7e2e-4c8b-9c10-058f04c2d359"). InnerVolumeSpecName "kube-api-access-vspfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:37:29.980445 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.980416    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vspfj\" (UniqueName: \"kubernetes.io/projected/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-kube-api-access-vspfj\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:29.980445 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.980443    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:29.980626 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:29.980455    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3ca3153e-7e2e-4c8b-9c10-058f04c2d359-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:30.044135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.044060    2565 generic.go:356] "Generic (PLEG): container finished" podID="3ca3153e-7e2e-4c8b-9c10-058f04c2d359" containerID="107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80" exitCode=0
Sep 17 14:37:30.044135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.044103    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"
Sep 17 14:37:30.044135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.044114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx" event={"ID":"3ca3153e-7e2e-4c8b-9c10-058f04c2d359","Type":"ContainerDied","Data":"107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80"}
Sep 17 14:37:30.044413 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.044144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx" event={"ID":"3ca3153e-7e2e-4c8b-9c10-058f04c2d359","Type":"ContainerDied","Data":"5954d904b200f00246397c0cc2c5bb6246034425f0ca7892974ac02bfceccc44"}
Sep 17 14:37:30.044413 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.044162    2565 scope.go:117] "RemoveContainer" containerID="107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80"
Sep 17 14:37:30.053998 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.053977    2565 scope.go:117] "RemoveContainer" containerID="107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80"
Sep 17 14:37:30.054323 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:37:30.054300    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80\": container with ID starting with 107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80 not found: ID does not exist" containerID="107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80"
Sep 17 14:37:30.054424 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.054332    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80"} err="failed to get container status \"107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80\": rpc error: code = NotFound desc = could not find container \"107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80\": container with ID starting with 107eb730d34347c324f7879842edfa3b1ad87f8e005503428e43fff6ac286d80 not found: ID does not exist"
Sep 17 14:37:30.067034 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.067008    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"]
Sep 17 14:37:30.070530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.070508    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/auth-test-model-3da4be-kserve-5f564b7747-qgphx"]
Sep 17 14:37:30.416511 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.416440    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt" podUID="38aefff7-cf34-474b-9b39-c6a10d1390d3" containerName="main" containerID="cri-o://506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03" gracePeriod=45
Sep 17 14:37:30.679603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.679576    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:37:30.787733 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.787699    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-dshm\") pod \"38aefff7-cf34-474b-9b39-c6a10d1390d3\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") "
Sep 17 14:37:30.787919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.787747    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5dtn\" (UniqueName: \"kubernetes.io/projected/38aefff7-cf34-474b-9b39-c6a10d1390d3-kube-api-access-h5dtn\") pod \"38aefff7-cf34-474b-9b39-c6a10d1390d3\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") "
Sep 17 14:37:30.787919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.787774    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-home\") pod \"38aefff7-cf34-474b-9b39-c6a10d1390d3\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") "
Sep 17 14:37:30.787919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.787810    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38aefff7-cf34-474b-9b39-c6a10d1390d3-tls-certs\") pod \"38aefff7-cf34-474b-9b39-c6a10d1390d3\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") "
Sep 17 14:37:30.787919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.787885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-model-cache\") pod \"38aefff7-cf34-474b-9b39-c6a10d1390d3\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") "
Sep 17 14:37:30.788156 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.787925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-tmp-dir\") pod \"38aefff7-cf34-474b-9b39-c6a10d1390d3\" (UID: \"38aefff7-cf34-474b-9b39-c6a10d1390d3\") "
Sep 17 14:37:30.788156 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.788078    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-home" (OuterVolumeSpecName: "home") pod "38aefff7-cf34-474b-9b39-c6a10d1390d3" (UID: "38aefff7-cf34-474b-9b39-c6a10d1390d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:37:30.788282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.788210    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:30.788282 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.788208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-model-cache" (OuterVolumeSpecName: "model-cache") pod "38aefff7-cf34-474b-9b39-c6a10d1390d3" (UID: "38aefff7-cf34-474b-9b39-c6a10d1390d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:37:30.788372 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.788298    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "38aefff7-cf34-474b-9b39-c6a10d1390d3" (UID: "38aefff7-cf34-474b-9b39-c6a10d1390d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:37:30.790023 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.790004    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38aefff7-cf34-474b-9b39-c6a10d1390d3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "38aefff7-cf34-474b-9b39-c6a10d1390d3" (UID: "38aefff7-cf34-474b-9b39-c6a10d1390d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:37:30.790328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.790300    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38aefff7-cf34-474b-9b39-c6a10d1390d3-kube-api-access-h5dtn" (OuterVolumeSpecName: "kube-api-access-h5dtn") pod "38aefff7-cf34-474b-9b39-c6a10d1390d3" (UID: "38aefff7-cf34-474b-9b39-c6a10d1390d3"). InnerVolumeSpecName "kube-api-access-h5dtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:37:30.790328 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.790308    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-dshm" (OuterVolumeSpecName: "dshm") pod "38aefff7-cf34-474b-9b39-c6a10d1390d3" (UID: "38aefff7-cf34-474b-9b39-c6a10d1390d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:37:30.889213 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.889177    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/38aefff7-cf34-474b-9b39-c6a10d1390d3-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:30.889213 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.889206    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:30.889213 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.889215    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:30.889463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.889223    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/38aefff7-cf34-474b-9b39-c6a10d1390d3-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:30.889463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:30.889231    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5dtn\" (UniqueName: \"kubernetes.io/projected/38aefff7-cf34-474b-9b39-c6a10d1390d3-kube-api-access-h5dtn\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:37:31.053564 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.053467    2565 generic.go:356] "Generic (PLEG): container finished" podID="38aefff7-cf34-474b-9b39-c6a10d1390d3" containerID="506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03" exitCode=0
Sep 17 14:37:31.053564 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.053518    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"
Sep 17 14:37:31.053775 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.053562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt" event={"ID":"38aefff7-cf34-474b-9b39-c6a10d1390d3","Type":"ContainerDied","Data":"506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03"}
Sep 17 14:37:31.053775 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.053601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt" event={"ID":"38aefff7-cf34-474b-9b39-c6a10d1390d3","Type":"ContainerDied","Data":"1eeb0474a9af699e378252505268f090dc06fb2f3e4ce00cf1a1b482f8d5240e"}
Sep 17 14:37:31.053775 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.053619    2565 scope.go:117] "RemoveContainer" containerID="506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03"
Sep 17 14:37:31.063419 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.063401    2565 scope.go:117] "RemoveContainer" containerID="506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03"
Sep 17 14:37:31.063706 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:37:31.063686    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03\": container with ID starting with 506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03 not found: ID does not exist" containerID="506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03"
Sep 17 14:37:31.063755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.063714    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03"} err="failed to get container status \"506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03\": rpc error: code = NotFound desc = could not find container \"506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03\": container with ID starting with 506d807ce5b044687371967e07505272ce645a34c8c7e3f2dc1fab5dd3ec8c03 not found: ID does not exist"
Sep 17 14:37:31.075718 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.075693    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"]
Sep 17 14:37:31.079547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.079525    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/auth-test-model-b48b7c-kserve-68dd67f84c-nn8lt"]
Sep 17 14:37:31.520101 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.520062    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38aefff7-cf34-474b-9b39-c6a10d1390d3" path="/var/lib/kubelet/pods/38aefff7-cf34-474b-9b39-c6a10d1390d3/volumes"
Sep 17 14:37:31.520539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:31.520523    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ca3153e-7e2e-4c8b-9c10-058f04c2d359" path="/var/lib/kubelet/pods/3ca3153e-7e2e-4c8b-9c10-058f04c2d359/volumes"
Sep 17 14:37:35.716337 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716302    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"]
Sep 17 14:37:35.716746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716731    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38aefff7-cf34-474b-9b39-c6a10d1390d3" containerName="main"
Sep 17 14:37:35.716794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716749    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="38aefff7-cf34-474b-9b39-c6a10d1390d3" containerName="main"
Sep 17 14:37:35.716794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716760    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88844dda-dea8-4f9f-9fa2-8aac82f87cf2" containerName="revoke-keys"
Sep 17 14:37:35.716794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716768    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88844dda-dea8-4f9f-9fa2-8aac82f87cf2" containerName="revoke-keys"
Sep 17 14:37:35.716794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716775    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" containerName="main"
Sep 17 14:37:35.716794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716780    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" containerName="main"
Sep 17 14:37:35.716794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716787    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy"
Sep 17 14:37:35.716794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716792    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716804    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ca3153e-7e2e-4c8b-9c10-058f04c2d359" containerName="main"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716810    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ca3153e-7e2e-4c8b-9c10-058f04c2d359" containerName="main"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716829    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f89105-a7b9-4f2b-b445-0dc45f5a06de" containerName="main"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716835    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f89105-a7b9-4f2b-b445-0dc45f5a06de" containerName="main"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716898    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716907    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ca3153e-7e2e-4c8b-9c10-058f04c2d359" containerName="main"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716915    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716925    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88844dda-dea8-4f9f-9fa2-8aac82f87cf2" containerName="revoke-keys"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716931    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9f89105-a7b9-4f2b-b445-0dc45f5a06de" containerName="main"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716936    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59b729d-1f73-4ac4-a9a8-9bc05f32e70b" containerName="main"
Sep 17 14:37:35.717029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.716942    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="38aefff7-cf34-474b-9b39-c6a10d1390d3" containerName="main"
Sep 17 14:37:35.720923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.720902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.724169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.724150    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"default-dockercfg-8klfk\""
Sep 17 14:37:35.724285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.724150    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"sub-test-model-3da4be-kserve-self-signed-certs\""
Sep 17 14:37:35.724285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.724201    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"openshift-service-ca.crt\""
Sep 17 14:37:35.724285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.724180    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"kube-root-ca.crt\""
Sep 17 14:37:35.729971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.729946    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-home\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.730069 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.730029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2280c06-599c-47ea-84d5-f5909d19d7d9-tls-certs\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.730165 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.730147    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-model-cache\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.730205 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.730172    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-tmp-dir\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.730240 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.730220    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-dshm\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.730303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.730238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dbpw\" (UniqueName: \"kubernetes.io/projected/c2280c06-599c-47ea-84d5-f5909d19d7d9-kube-api-access-4dbpw\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.734843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.734810    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"]
Sep 17 14:37:35.772736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.772691    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:35.831662 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.831612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2280c06-599c-47ea-84d5-f5909d19d7d9-tls-certs\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.831878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.831717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-model-cache\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.831878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.831742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-tmp-dir\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.831878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.831785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-dshm\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.831878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.831814    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dbpw\" (UniqueName: \"kubernetes.io/projected/c2280c06-599c-47ea-84d5-f5909d19d7d9-kube-api-access-4dbpw\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.831878 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.831843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-home\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.832278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.832224    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-model-cache\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.832368 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.832338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-tmp-dir\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.832435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.832369    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-home\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.834042 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.834019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-dshm\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.834452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.834432    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2280c06-599c-47ea-84d5-f5909d19d7d9-tls-certs\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:35.841849 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:35.841826    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dbpw\" (UniqueName: \"kubernetes.io/projected/c2280c06-599c-47ea-84d5-f5909d19d7d9-kube-api-access-4dbpw\") pod \"sub-test-model-3da4be-kserve-669f659cb5-cb9cp\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:36.030871 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:36.030749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:36.307318 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:36.307289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"]
Sep 17 14:37:36.309796 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:37:36.309769    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc2280c06_599c_47ea_84d5_f5909d19d7d9.slice/crio-7ee23f1ef0631ddbe2dbab98ac0cb5aff68586466ef5f82be3315b468a8ae0a2 WatchSource:0}: Error finding container 7ee23f1ef0631ddbe2dbab98ac0cb5aff68586466ef5f82be3315b468a8ae0a2: Status 404 returned error can't find the container with id 7ee23f1ef0631ddbe2dbab98ac0cb5aff68586466ef5f82be3315b468a8ae0a2
Sep 17 14:37:37.088799 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:37.088758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp" event={"ID":"c2280c06-599c-47ea-84d5-f5909d19d7d9","Type":"ContainerStarted","Data":"4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54"}
Sep 17 14:37:37.088799 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:37.088802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp" event={"ID":"c2280c06-599c-47ea-84d5-f5909d19d7d9","Type":"ContainerStarted","Data":"7ee23f1ef0631ddbe2dbab98ac0cb5aff68586466ef5f82be3315b468a8ae0a2"}
Sep 17 14:37:37.108480 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:37.108427    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp" podStartSLOduration=2.108410796 podStartE2EDuration="2.108410796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:37:37.106527116 +0000 UTC m=+1844.040032309" watchObservedRunningTime="2026-09-17 14:37:37.108410796 +0000 UTC m=+1844.041915988"
Sep 17 14:37:40.945220 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:40.945187    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:46.031798 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:46.031762    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:46.032210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:46.031820    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:46.044547 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:46.044520    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:46.150879 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:46.150850    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:37:54.037550 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:54.037518    2565 scope.go:117] "RemoveContainer" containerID="5da15f8a0273dc31ab5deead45a3c7b321b3f264289eaf255ff8295eef1e4fc0"
Sep 17 14:37:56.817100 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.817067    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"]
Sep 17 14:37:56.817576 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.817505    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy"
Sep 17 14:37:56.817576 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.817517    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6301c8c7-7838-4cc3-b149-8e1c506e3da7" containerName="istio-proxy"
Sep 17 14:37:56.819498 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.819469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:56.821617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.821585    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"openshift-service-ca.crt\""
Sep 17 14:37:56.822371 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.822347    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"default-dockercfg-7ltgp\""
Sep 17 14:37:56.822603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.822401    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"kube-root-ca.crt\""
Sep 17 14:37:56.822910 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.822455    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"sub-test-model-b48b7c-kserve-self-signed-certs\""
Sep 17 14:37:56.829790 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.829765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"]
Sep 17 14:37:56.847959 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.847918    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:37:56.938368 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.938335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tls-certs\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:56.938521 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.938450    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tmp-dir\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:56.938577 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.938522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brspp\" (UniqueName: \"kubernetes.io/projected/b0cc5f2f-845c-4183-8591-0c566dba2eb7-kube-api-access-brspp\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:56.938577 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.938546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-home\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:56.938651 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.938613    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-model-cache\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:56.938651 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:56.938638    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-dshm\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tmp-dir\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brspp\" (UniqueName: \"kubernetes.io/projected/b0cc5f2f-845c-4183-8591-0c566dba2eb7-kube-api-access-brspp\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-home\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-model-cache\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-dshm\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tls-certs\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040609    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tmp-dir\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-model-cache\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.040770 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.040662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-home\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.042616 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.042596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-dshm\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.042808 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.042790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tls-certs\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.050111 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.050084    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brspp\" (UniqueName: \"kubernetes.io/projected/b0cc5f2f-845c-4183-8591-0c566dba2eb7-kube-api-access-brspp\") pod \"sub-test-model-b48b7c-kserve-b69bc7699-cgdfq\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.130404 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.130366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:37:57.400025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:57.399997    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"]
Sep 17 14:37:57.401809 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:37:57.401767    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0cc5f2f_845c_4183_8591_0c566dba2eb7.slice/crio-1a92936171cae8cc6fede074dda2a2516ad07dac2615bbf9b1bb35ad47965386 WatchSource:0}: Error finding container 1a92936171cae8cc6fede074dda2a2516ad07dac2615bbf9b1bb35ad47965386: Status 404 returned error can't find the container with id 1a92936171cae8cc6fede074dda2a2516ad07dac2615bbf9b1bb35ad47965386
Sep 17 14:37:58.204831 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:58.204788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq" event={"ID":"b0cc5f2f-845c-4183-8591-0c566dba2eb7","Type":"ContainerStarted","Data":"ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777"}
Sep 17 14:37:58.204831 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:58.204837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq" event={"ID":"b0cc5f2f-845c-4183-8591-0c566dba2eb7","Type":"ContainerStarted","Data":"1a92936171cae8cc6fede074dda2a2516ad07dac2615bbf9b1bb35ad47965386"}
Sep 17 14:37:58.221357 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:37:58.221307    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq" podStartSLOduration=2.221290901 podStartE2EDuration="2.221290901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:37:58.219803441 +0000 UTC m=+1865.153308635" watchObservedRunningTime="2026-09-17 14:37:58.221290901 +0000 UTC m=+1865.154796092"
Sep 17 14:38:07.130824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:07.130784    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:38:07.131352 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:07.130844    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:38:07.143044 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:07.143016    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:38:07.264346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:07.264318    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:38:35.351753 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:35.351715    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:35.452096 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:35.452056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:35.623138 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:35.623046    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"]
Sep 17 14:38:35.646509 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:35.646474    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:36.584454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:36.584415    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"]
Sep 17 14:38:36.599001 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:36.598975    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:37.121975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:37.121940    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:37.223685 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:37.223619    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:38:50.642729 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.642667    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp" podUID="c2280c06-599c-47ea-84d5-f5909d19d7d9" containerName="main" containerID="cri-o://4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54" gracePeriod=45
Sep 17 14:38:50.894128 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.894070    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:38:50.979092 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979064    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2280c06-599c-47ea-84d5-f5909d19d7d9-tls-certs\") pod \"c2280c06-599c-47ea-84d5-f5909d19d7d9\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") "
Sep 17 14:38:50.979242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979112    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-dshm\") pod \"c2280c06-599c-47ea-84d5-f5909d19d7d9\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") "
Sep 17 14:38:50.979242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979171    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dbpw\" (UniqueName: \"kubernetes.io/projected/c2280c06-599c-47ea-84d5-f5909d19d7d9-kube-api-access-4dbpw\") pod \"c2280c06-599c-47ea-84d5-f5909d19d7d9\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") "
Sep 17 14:38:50.979242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979201    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-model-cache\") pod \"c2280c06-599c-47ea-84d5-f5909d19d7d9\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") "
Sep 17 14:38:50.979242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979222    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-home\") pod \"c2280c06-599c-47ea-84d5-f5909d19d7d9\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") "
Sep 17 14:38:50.979470 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979286    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-tmp-dir\") pod \"c2280c06-599c-47ea-84d5-f5909d19d7d9\" (UID: \"c2280c06-599c-47ea-84d5-f5909d19d7d9\") "
Sep 17 14:38:50.979532 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979514    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c2280c06-599c-47ea-84d5-f5909d19d7d9" (UID: "c2280c06-599c-47ea-84d5-f5909d19d7d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:38:50.979591 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979531    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-home" (OuterVolumeSpecName: "home") pod "c2280c06-599c-47ea-84d5-f5909d19d7d9" (UID: "c2280c06-599c-47ea-84d5-f5909d19d7d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:38:50.979648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979612    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c2280c06-599c-47ea-84d5-f5909d19d7d9" (UID: "c2280c06-599c-47ea-84d5-f5909d19d7d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:38:50.979726 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979708    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:50.979769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979732    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:50.979769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.979745    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:50.981044 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.981022    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-dshm" (OuterVolumeSpecName: "dshm") pod "c2280c06-599c-47ea-84d5-f5909d19d7d9" (UID: "c2280c06-599c-47ea-84d5-f5909d19d7d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:38:50.981178 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.981161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2280c06-599c-47ea-84d5-f5909d19d7d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c2280c06-599c-47ea-84d5-f5909d19d7d9" (UID: "c2280c06-599c-47ea-84d5-f5909d19d7d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:38:50.981240 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:50.981198    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2280c06-599c-47ea-84d5-f5909d19d7d9-kube-api-access-4dbpw" (OuterVolumeSpecName: "kube-api-access-4dbpw") pod "c2280c06-599c-47ea-84d5-f5909d19d7d9" (UID: "c2280c06-599c-47ea-84d5-f5909d19d7d9"). InnerVolumeSpecName "kube-api-access-4dbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:38:51.080606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.080573    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c2280c06-599c-47ea-84d5-f5909d19d7d9-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:51.080606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.080601    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c2280c06-599c-47ea-84d5-f5909d19d7d9-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:51.080606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.080611    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dbpw\" (UniqueName: \"kubernetes.io/projected/c2280c06-599c-47ea-84d5-f5909d19d7d9-kube-api-access-4dbpw\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:51.496606 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.496572    2565 generic.go:356] "Generic (PLEG): container finished" podID="c2280c06-599c-47ea-84d5-f5909d19d7d9" containerID="4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54" exitCode=0
Sep 17 14:38:51.496773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.496613    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"
Sep 17 14:38:51.496773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.496616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp" event={"ID":"c2280c06-599c-47ea-84d5-f5909d19d7d9","Type":"ContainerDied","Data":"4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54"}
Sep 17 14:38:51.496773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.496739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp" event={"ID":"c2280c06-599c-47ea-84d5-f5909d19d7d9","Type":"ContainerDied","Data":"7ee23f1ef0631ddbe2dbab98ac0cb5aff68586466ef5f82be3315b468a8ae0a2"}
Sep 17 14:38:51.496773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.496763    2565 scope.go:117] "RemoveContainer" containerID="4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54"
Sep 17 14:38:51.506448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.506433    2565 scope.go:117] "RemoveContainer" containerID="4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54"
Sep 17 14:38:51.506687 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:38:51.506671    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54\": container with ID starting with 4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54 not found: ID does not exist" containerID="4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54"
Sep 17 14:38:51.506732 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.506695    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54"} err="failed to get container status \"4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54\": rpc error: code = NotFound desc = could not find container \"4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54\": container with ID starting with 4b6490f8bf9c1c95f0bbcf85d57462d3feeaad5558894ec4d44cbcc1898eec54 not found: ID does not exist"
Sep 17 14:38:51.520208 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.520187    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"]
Sep 17 14:38:51.521675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.521658    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/sub-test-model-3da4be-kserve-669f659cb5-cb9cp"]
Sep 17 14:38:51.604923 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.604879    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq" podUID="b0cc5f2f-845c-4183-8591-0c566dba2eb7" containerName="main" containerID="cri-o://ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777" gracePeriod=45
Sep 17 14:38:51.865177 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.865150    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:38:51.987944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.987911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tls-certs\") pod \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") "
Sep 17 14:38:51.987944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.987948    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-dshm\") pod \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") "
Sep 17 14:38:51.988151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.987979    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brspp\" (UniqueName: \"kubernetes.io/projected/b0cc5f2f-845c-4183-8591-0c566dba2eb7-kube-api-access-brspp\") pod \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") "
Sep 17 14:38:51.988151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988116    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-model-cache\") pod \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") "
Sep 17 14:38:51.988276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tmp-dir\") pod \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") "
Sep 17 14:38:51.988276 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-home\") pod \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\" (UID: \"b0cc5f2f-845c-4183-8591-0c566dba2eb7\") "
Sep 17 14:38:51.988428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988398    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b0cc5f2f-845c-4183-8591-0c566dba2eb7" (UID: "b0cc5f2f-845c-4183-8591-0c566dba2eb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:38:51.988549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988500    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b0cc5f2f-845c-4183-8591-0c566dba2eb7" (UID: "b0cc5f2f-845c-4183-8591-0c566dba2eb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:38:51.988633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988541    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-home" (OuterVolumeSpecName: "home") pod "b0cc5f2f-845c-4183-8591-0c566dba2eb7" (UID: "b0cc5f2f-845c-4183-8591-0c566dba2eb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:38:51.988679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988637    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:51.988679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988655    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:51.988679 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.988667    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:51.990279 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.990228    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-dshm" (OuterVolumeSpecName: "dshm") pod "b0cc5f2f-845c-4183-8591-0c566dba2eb7" (UID: "b0cc5f2f-845c-4183-8591-0c566dba2eb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:38:51.990375 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.990278    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0cc5f2f-845c-4183-8591-0c566dba2eb7-kube-api-access-brspp" (OuterVolumeSpecName: "kube-api-access-brspp") pod "b0cc5f2f-845c-4183-8591-0c566dba2eb7" (UID: "b0cc5f2f-845c-4183-8591-0c566dba2eb7"). InnerVolumeSpecName "kube-api-access-brspp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:38:51.990375 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:51.990313    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b0cc5f2f-845c-4183-8591-0c566dba2eb7" (UID: "b0cc5f2f-845c-4183-8591-0c566dba2eb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:38:52.089339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.089304    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0cc5f2f-845c-4183-8591-0c566dba2eb7-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:52.089339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.089332    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b0cc5f2f-845c-4183-8591-0c566dba2eb7-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:52.089339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.089342    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brspp\" (UniqueName: \"kubernetes.io/projected/b0cc5f2f-845c-4183-8591-0c566dba2eb7-kube-api-access-brspp\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:38:52.503507 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.503410    2565 generic.go:356] "Generic (PLEG): container finished" podID="b0cc5f2f-845c-4183-8591-0c566dba2eb7" containerID="ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777" exitCode=0
Sep 17 14:38:52.503507 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.503451    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"
Sep 17 14:38:52.503507 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.503494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq" event={"ID":"b0cc5f2f-845c-4183-8591-0c566dba2eb7","Type":"ContainerDied","Data":"ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777"}
Sep 17 14:38:52.503805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.503530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq" event={"ID":"b0cc5f2f-845c-4183-8591-0c566dba2eb7","Type":"ContainerDied","Data":"1a92936171cae8cc6fede074dda2a2516ad07dac2615bbf9b1bb35ad47965386"}
Sep 17 14:38:52.503805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.503548    2565 scope.go:117] "RemoveContainer" containerID="ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777"
Sep 17 14:38:52.512689 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.512672    2565 scope.go:117] "RemoveContainer" containerID="ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777"
Sep 17 14:38:52.512976 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:38:52.512957    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777\": container with ID starting with ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777 not found: ID does not exist" containerID="ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777"
Sep 17 14:38:52.513060 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.512982    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777"} err="failed to get container status \"ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777\": rpc error: code = NotFound desc = could not find container \"ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777\": container with ID starting with ee3c8d8e5b1da3458bccd1fef764e9fa6a36d02bb8dabd77c03b107472312777 not found: ID does not exist"
Sep 17 14:38:52.525106 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.525083    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"]
Sep 17 14:38:52.529333 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:52.529311    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/sub-test-model-b48b7c-kserve-b69bc7699-cgdfq"]
Sep 17 14:38:53.520990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:53.520957    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0cc5f2f-845c-4183-8591-0c566dba2eb7" path="/var/lib/kubelet/pods/b0cc5f2f-845c-4183-8591-0c566dba2eb7/volumes"
Sep 17 14:38:53.521412 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:38:53.521396    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2280c06-599c-47ea-84d5-f5909d19d7d9" path="/var/lib/kubelet/pods/c2280c06-599c-47ea-84d5-f5909d19d7d9/volumes"
Sep 17 14:39:03.617375 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.617336    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"]
Sep 17 14:39:03.617901 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.617877    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2280c06-599c-47ea-84d5-f5909d19d7d9" containerName="main"
Sep 17 14:39:03.617957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.617905    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2280c06-599c-47ea-84d5-f5909d19d7d9" containerName="main"
Sep 17 14:39:03.617957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.617947    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0cc5f2f-845c-4183-8591-0c566dba2eb7" containerName="main"
Sep 17 14:39:03.617957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.617955    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0cc5f2f-845c-4183-8591-0c566dba2eb7" containerName="main"
Sep 17 14:39:03.618069 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.618061    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0cc5f2f-845c-4183-8591-0c566dba2eb7" containerName="main"
Sep 17 14:39:03.618110 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.618074    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2280c06-599c-47ea-84d5-f5909d19d7d9" containerName="main"
Sep 17 14:39:03.621624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.621594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.624039 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.624007    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"openshift-service-ca.crt\""
Sep 17 14:39:03.624899 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.624871    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"rate-test-model-3da4be-kserve-self-signed-certs\""
Sep 17 14:39:03.625093 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.624898    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"default-dockercfg-8klfk\""
Sep 17 14:39:03.625274 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.624942    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3da4be\"/\"kube-root-ca.crt\""
Sep 17 14:39:03.636767 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.636736    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"]
Sep 17 14:39:03.659824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.659786    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:39:03.703149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.703095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-model-cache\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.703149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.703141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tmp-dir\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.703448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.703181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvx2s\" (UniqueName: \"kubernetes.io/projected/ba929e2b-6263-4a5f-8c60-db8cfa94222f-kube-api-access-vvx2s\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.703448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.703272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-home\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.703448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.703314    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tls-certs\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.703448 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.703334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-dshm\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.803852 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.803815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvx2s\" (UniqueName: \"kubernetes.io/projected/ba929e2b-6263-4a5f-8c60-db8cfa94222f-kube-api-access-vvx2s\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.804055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.803873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-home\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.804055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.803905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tls-certs\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.804055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.803921    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-dshm\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.804055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.803982    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-model-cache\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.804055 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.804008    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tmp-dir\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.804385 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.804358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-home\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.804462 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.804423    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tmp-dir\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.804536 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.804517    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-model-cache\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.806353 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.806327    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-dshm\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.806647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.806631    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tls-certs\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.812573 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.812544    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvx2s\" (UniqueName: \"kubernetes.io/projected/ba929e2b-6263-4a5f-8c60-db8cfa94222f-kube-api-access-vvx2s\") pod \"rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") " pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:03.932134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:03.932040    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:04.210364 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:04.210335    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"]
Sep 17 14:39:04.212168 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:39:04.212125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba929e2b_6263_4a5f_8c60_db8cfa94222f.slice/crio-578ea7d31ff0b3354fd74bc491edd1fd61bd8d67ba273a9037a153712c6e025b WatchSource:0}: Error finding container 578ea7d31ff0b3354fd74bc491edd1fd61bd8d67ba273a9037a153712c6e025b: Status 404 returned error can't find the container with id 578ea7d31ff0b3354fd74bc491edd1fd61bd8d67ba273a9037a153712c6e025b
Sep 17 14:39:04.573607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:04.573575    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4" event={"ID":"ba929e2b-6263-4a5f-8c60-db8cfa94222f","Type":"ContainerStarted","Data":"91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d"}
Sep 17 14:39:04.573773 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:04.573616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4" event={"ID":"ba929e2b-6263-4a5f-8c60-db8cfa94222f","Type":"ContainerStarted","Data":"578ea7d31ff0b3354fd74bc491edd1fd61bd8d67ba273a9037a153712c6e025b"}
Sep 17 14:39:04.592008 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:04.591947    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4" podStartSLOduration=1.591929734 podStartE2EDuration="1.591929734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:39:04.590443963 +0000 UTC m=+1931.523949155" watchObservedRunningTime="2026-09-17 14:39:04.591929734 +0000 UTC m=+1931.525434954"
Sep 17 14:39:13.932590 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:13.932540    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:13.933059 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:13.932609    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:13.946201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:13.946171    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:14.635347 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:14.635317    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:39:27.617784 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.617751    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"]
Sep 17 14:39:27.621149 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.621130    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.624001 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.623977    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"kube-root-ca.crt\""
Sep 17 14:39:27.624134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.624041    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"openshift-service-ca.crt\""
Sep 17 14:39:27.624134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.623977    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"rate-test-model-b48b7c-kserve-self-signed-certs\""
Sep 17 14:39:27.624134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.624117    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-b48b7c\"/\"default-dockercfg-7ltgp\""
Sep 17 14:39:27.629802 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.629780    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"]
Sep 17 14:39:27.711575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.711541    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:39:27.740439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.740404    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-tmp-dir\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.740618 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.740464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae8c7b4-be75-496d-bc96-38981558ace0-tls-certs\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.740618 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.740522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phvjp\" (UniqueName: \"kubernetes.io/projected/cae8c7b4-be75-496d-bc96-38981558ace0-kube-api-access-phvjp\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.740618 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.740602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-model-cache\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.740755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.740708    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-dshm\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.740795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.740762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-home\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.842439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.842396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-model-cache\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.842625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.842514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-dshm\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.842625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.842571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-home\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.842625 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.842604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-tmp-dir\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.842815 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.842665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae8c7b4-be75-496d-bc96-38981558ace0-tls-certs\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.842815 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.842709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phvjp\" (UniqueName: \"kubernetes.io/projected/cae8c7b4-be75-496d-bc96-38981558ace0-kube-api-access-phvjp\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.842933 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.842816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-model-cache\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.843056 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.843026    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-home\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.843268 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.843232    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-tmp-dir\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.844886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.844858    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-dshm\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.845385 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.845364    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae8c7b4-be75-496d-bc96-38981558ace0-tls-certs\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.853819 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.853792    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phvjp\" (UniqueName: \"kubernetes.io/projected/cae8c7b4-be75-496d-bc96-38981558ace0-kube-api-access-phvjp\") pod \"rate-test-model-b48b7c-kserve-86bd48b59-nqqwz\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") " pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:27.932596 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:27.932515    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:28.208021 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:28.207996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"]
Sep 17 14:39:28.210122 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:39:28.210091    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcae8c7b4_be75_496d_bc96_38981558ace0.slice/crio-6e65f6a59e464b0d5025f61671689485a5f0ef30c082fbc0f1702a295f7d0e07 WatchSource:0}: Error finding container 6e65f6a59e464b0d5025f61671689485a5f0ef30c082fbc0f1702a295f7d0e07: Status 404 returned error can't find the container with id 6e65f6a59e464b0d5025f61671689485a5f0ef30c082fbc0f1702a295f7d0e07
Sep 17 14:39:28.701780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:28.701742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz" event={"ID":"cae8c7b4-be75-496d-bc96-38981558ace0","Type":"ContainerStarted","Data":"31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193"}
Sep 17 14:39:28.701780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:28.701783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz" event={"ID":"cae8c7b4-be75-496d-bc96-38981558ace0","Type":"ContainerStarted","Data":"6e65f6a59e464b0d5025f61671689485a5f0ef30c082fbc0f1702a295f7d0e07"}
Sep 17 14:39:28.722036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:28.721979    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz" podStartSLOduration=1.721963411 podStartE2EDuration="1.721963411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:39:28.72000732 +0000 UTC m=+1955.653512510" watchObservedRunningTime="2026-09-17 14:39:28.721963411 +0000 UTC m=+1955.655468602"
Sep 17 14:39:37.933437 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:37.933392    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:37.933856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:37.933456    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:37.946480 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:37.946452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:39:38.772161 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:39:38.772128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:40:02.910820 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:02.910736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"]
Sep 17 14:40:02.925586 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:02.925545    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:04.072630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:04.072598    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"]
Sep 17 14:40:04.105027 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:04.104990    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:04.795522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:04.795481    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:04.960698 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:04.960664    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"]
Sep 17 14:40:04.964564 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:04.964542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:04.986352 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:04.986315    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"]
Sep 17 14:40:04.993427 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:04.993400    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-c2db32-openshift-default-dockercfg-89svw\""
Sep 17 14:40:05.005822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.005788    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:05.014273 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014214    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-envoy\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.014435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-istio-token\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.014435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014329    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj7zz\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-kube-api-access-rj7zz\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.014435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c142b9-ad5b-4e45-8588-622475787825-istiod-ca-cert\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.014613 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014447    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-data\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.014613 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-credential-socket\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.014613 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-certs\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.014765 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c142b9-ad5b-4e45-8588-622475787825-istio-podinfo\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.014765 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.014689    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-socket\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115336 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115290    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-envoy\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-istio-token\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj7zz\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-kube-api-access-rj7zz\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115421    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c142b9-ad5b-4e45-8588-622475787825-istiod-ca-cert\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115453    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-data\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-credential-socket\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-certs\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c142b9-ad5b-4e45-8588-622475787825-istio-podinfo\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.115855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-socket\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.116182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115901    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-data\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.116182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.115921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-credential-socket\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.116182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.116031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-certs\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.116182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.116089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-socket\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.116346 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.116232    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c142b9-ad5b-4e45-8588-622475787825-istiod-ca-cert\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.117804 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.117781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-envoy\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.118165 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.118147    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c142b9-ad5b-4e45-8588-622475787825-istio-podinfo\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.126921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.126891    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj7zz\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-kube-api-access-rj7zz\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.127066 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.126998    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-istio-token\") pod \"e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") " pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.212185 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.212150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:05.275358 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.275298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:05.640025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.639995    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"]
Sep 17 14:40:05.643164 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:40:05.643124    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c142b9_ad5b_4e45_8588_622475787825.slice/crio-e4394d8cb146993c0d21f3a5ee573417fe3d70e489bca399e06ce22d4e54098f WatchSource:0}: Error finding container e4394d8cb146993c0d21f3a5ee573417fe3d70e489bca399e06ce22d4e54098f: Status 404 returned error can't find the container with id e4394d8cb146993c0d21f3a5ee573417fe3d70e489bca399e06ce22d4e54098f
Sep 17 14:40:05.645583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.645548    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:05.645677 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.645619    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:05.645677 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.645647    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:05.913132 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.913043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" event={"ID":"29c142b9-ad5b-4e45-8588-622475787825","Type":"ContainerStarted","Data":"72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9"}
Sep 17 14:40:05.913132 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.913105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" event={"ID":"29c142b9-ad5b-4e45-8588-622475787825","Type":"ContainerStarted","Data":"e4394d8cb146993c0d21f3a5ee573417fe3d70e489bca399e06ce22d4e54098f"}
Sep 17 14:40:05.952197 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:05.952128    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podStartSLOduration=1.952104198 podStartE2EDuration="1.952104198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:05.948078395 +0000 UTC m=+1992.881583599" watchObservedRunningTime="2026-09-17 14:40:05.952104198 +0000 UTC m=+1992.885609395"
Sep 17 14:40:06.275772 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:06.275687    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:06.281123 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:06.281095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:06.918305 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:06.918275    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:06.919385 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:06.919360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:07.764076 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:07.764044    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:07.939697 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:07.939662    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:08.038868 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:08.038771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:08.575824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:08.575793    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:09.567451 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:09.567412    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:09.665512 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:09.665478    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:10.083140 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.083108    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:10.179881 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.179852    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:10.231715 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.231678    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"]
Sep 17 14:40:10.235361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.235336    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.239081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.239054    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c2db32\""
Sep 17 14:40:10.239233 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.239210    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c2db32-dockercfg-szr2z\""
Sep 17 14:40:10.248112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.248090    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"]
Sep 17 14:40:10.267158 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.267128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e56d6f1-2b53-4194-89cb-76dcc45dd282-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds\" (UID: \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.267312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.267192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phkzh\" (UniqueName: \"kubernetes.io/projected/4e56d6f1-2b53-4194-89cb-76dcc45dd282-kube-api-access-phkzh\") pod \"payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds\" (UID: \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.274084 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.274057    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"]
Sep 17 14:40:10.277613 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.277596    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.293930 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.293898    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"]
Sep 17 14:40:10.368036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.367954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phkzh\" (UniqueName: \"kubernetes.io/projected/4e56d6f1-2b53-4194-89cb-76dcc45dd282-kube-api-access-phkzh\") pod \"payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds\" (UID: \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.368036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.368009    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lb8x\" (UniqueName: \"kubernetes.io/projected/88f7d27c-3414-4a45-8773-1a9482ea3007-kube-api-access-6lb8x\") pod \"payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8\" (UID: \"88f7d27c-3414-4a45-8773-1a9482ea3007\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.368230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.368083    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f7d27c-3414-4a45-8773-1a9482ea3007-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8\" (UID: \"88f7d27c-3414-4a45-8773-1a9482ea3007\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.368230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.368153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e56d6f1-2b53-4194-89cb-76dcc45dd282-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds\" (UID: \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.368864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.368836    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e56d6f1-2b53-4194-89cb-76dcc45dd282-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds\" (UID: \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.376797 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.376773    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phkzh\" (UniqueName: \"kubernetes.io/projected/4e56d6f1-2b53-4194-89cb-76dcc45dd282-kube-api-access-phkzh\") pod \"payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds\" (UID: \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.468824 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.468778    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f7d27c-3414-4a45-8773-1a9482ea3007-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8\" (UID: \"88f7d27c-3414-4a45-8773-1a9482ea3007\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.468982 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.468888    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lb8x\" (UniqueName: \"kubernetes.io/projected/88f7d27c-3414-4a45-8773-1a9482ea3007-kube-api-access-6lb8x\") pod \"payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8\" (UID: \"88f7d27c-3414-4a45-8773-1a9482ea3007\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.469537 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.469516    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f7d27c-3414-4a45-8773-1a9482ea3007-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8\" (UID: \"88f7d27c-3414-4a45-8773-1a9482ea3007\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.477194 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.477167    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lb8x\" (UniqueName: \"kubernetes.io/projected/88f7d27c-3414-4a45-8773-1a9482ea3007-kube-api-access-6lb8x\") pod \"payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8\" (UID: \"88f7d27c-3414-4a45-8773-1a9482ea3007\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.547170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.547132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.587924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.587887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.837053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.837029    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"]
Sep 17 14:40:10.839226 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:40:10.839197    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e56d6f1_2b53_4194_89cb_76dcc45dd282.slice/crio-6f7fcffb918d9fe1e790dd8c282f581424645c12af4dae9e5f7396246536e2d1 WatchSource:0}: Error finding container 6f7fcffb918d9fe1e790dd8c282f581424645c12af4dae9e5f7396246536e2d1: Status 404 returned error can't find the container with id 6f7fcffb918d9fe1e790dd8c282f581424645c12af4dae9e5f7396246536e2d1
Sep 17 14:40:10.870366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.870341    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"]
Sep 17 14:40:10.879396 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:40:10.879372    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f7d27c_3414_4a45_8773_1a9482ea3007.slice/crio-f83002941fa2f2c420660daa1bd4701ca2c9d108f45b8bcd727d8a8e115fdd36 WatchSource:0}: Error finding container f83002941fa2f2c420660daa1bd4701ca2c9d108f45b8bcd727d8a8e115fdd36: Status 404 returned error can't find the container with id f83002941fa2f2c420660daa1bd4701ca2c9d108f45b8bcd727d8a8e115fdd36
Sep 17 14:40:10.945883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.945612    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds" event={"ID":"4e56d6f1-2b53-4194-89cb-76dcc45dd282","Type":"ContainerStarted","Data":"752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f"}
Sep 17 14:40:10.945883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.945684    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds" event={"ID":"4e56d6f1-2b53-4194-89cb-76dcc45dd282","Type":"ContainerStarted","Data":"6f7fcffb918d9fe1e790dd8c282f581424645c12af4dae9e5f7396246536e2d1"}
Sep 17 14:40:10.946077 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.945904    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:10.947855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.947814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8" event={"ID":"88f7d27c-3414-4a45-8773-1a9482ea3007","Type":"ContainerStarted","Data":"03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db"}
Sep 17 14:40:10.947993 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.947866    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8" event={"ID":"88f7d27c-3414-4a45-8773-1a9482ea3007","Type":"ContainerStarted","Data":"f83002941fa2f2c420660daa1bd4701ca2c9d108f45b8bcd727d8a8e115fdd36"}
Sep 17 14:40:10.947993 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.947954    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:10.963811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.963745    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds" podStartSLOduration=0.963723967 podStartE2EDuration="963.723967ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:10.962669215 +0000 UTC m=+1997.896174448" watchObservedRunningTime="2026-09-17 14:40:10.963723967 +0000 UTC m=+1997.897229161"
Sep 17 14:40:10.980312 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:10.980240    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8" podStartSLOduration=0.980222456 podStartE2EDuration="980.222456ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:40:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:10.978991772 +0000 UTC m=+1997.912496964" watchObservedRunningTime="2026-09-17 14:40:10.980222456 +0000 UTC m=+1997.913727751"
Sep 17 14:40:12.866681 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:12.866641    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:17.929364 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:17.929328    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4" podUID="ba929e2b-6263-4a5f-8c60-db8cfa94222f" containerName="main" containerID="cri-o://91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d" gracePeriod=45
Sep 17 14:40:18.237664 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.237635    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:40:18.373278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.372642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tmp-dir\") pod \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") "
Sep 17 14:40:18.373278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.372704    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-model-cache\") pod \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") "
Sep 17 14:40:18.373278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.372744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-dshm\") pod \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") "
Sep 17 14:40:18.373278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.372842    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tls-certs\") pod \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") "
Sep 17 14:40:18.373278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.372886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvx2s\" (UniqueName: \"kubernetes.io/projected/ba929e2b-6263-4a5f-8c60-db8cfa94222f-kube-api-access-vvx2s\") pod \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") "
Sep 17 14:40:18.373278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.372912    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-home\") pod \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\" (UID: \"ba929e2b-6263-4a5f-8c60-db8cfa94222f\") "
Sep 17 14:40:18.373795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.373578    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-home" (OuterVolumeSpecName: "home") pod "ba929e2b-6263-4a5f-8c60-db8cfa94222f" (UID: "ba929e2b-6263-4a5f-8c60-db8cfa94222f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:40:18.373795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.373777    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ba929e2b-6263-4a5f-8c60-db8cfa94222f" (UID: "ba929e2b-6263-4a5f-8c60-db8cfa94222f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:40:18.375745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.373952    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-model-cache" (OuterVolumeSpecName: "model-cache") pod "ba929e2b-6263-4a5f-8c60-db8cfa94222f" (UID: "ba929e2b-6263-4a5f-8c60-db8cfa94222f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:40:18.379009 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.378326    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-dshm" (OuterVolumeSpecName: "dshm") pod "ba929e2b-6263-4a5f-8c60-db8cfa94222f" (UID: "ba929e2b-6263-4a5f-8c60-db8cfa94222f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:40:18.379009 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.378606    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba929e2b-6263-4a5f-8c60-db8cfa94222f-kube-api-access-vvx2s" (OuterVolumeSpecName: "kube-api-access-vvx2s") pod "ba929e2b-6263-4a5f-8c60-db8cfa94222f" (UID: "ba929e2b-6263-4a5f-8c60-db8cfa94222f"). InnerVolumeSpecName "kube-api-access-vvx2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:40:18.379009 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.378960    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ba929e2b-6263-4a5f-8c60-db8cfa94222f" (UID: "ba929e2b-6263-4a5f-8c60-db8cfa94222f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:40:18.475339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.474326    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:18.475339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.474373    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvx2s\" (UniqueName: \"kubernetes.io/projected/ba929e2b-6263-4a5f-8c60-db8cfa94222f-kube-api-access-vvx2s\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:18.475339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.474388    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:18.475339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.474401    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:18.475339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.474415    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:18.475339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:18.474427    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba929e2b-6263-4a5f-8c60-db8cfa94222f-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:19.004118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.004084    2565 generic.go:356] "Generic (PLEG): container finished" podID="ba929e2b-6263-4a5f-8c60-db8cfa94222f" containerID="91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d" exitCode=0
Sep 17 14:40:19.004546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.004126    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"
Sep 17 14:40:19.004546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.004175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4" event={"ID":"ba929e2b-6263-4a5f-8c60-db8cfa94222f","Type":"ContainerDied","Data":"91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d"}
Sep 17 14:40:19.004546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.004217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4" event={"ID":"ba929e2b-6263-4a5f-8c60-db8cfa94222f","Type":"ContainerDied","Data":"578ea7d31ff0b3354fd74bc491edd1fd61bd8d67ba273a9037a153712c6e025b"}
Sep 17 14:40:19.004546 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.004237    2565 scope.go:117] "RemoveContainer" containerID="91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d"
Sep 17 14:40:19.013646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.013629    2565 scope.go:117] "RemoveContainer" containerID="91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d"
Sep 17 14:40:19.013915 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:40:19.013890    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d\": container with ID starting with 91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d not found: ID does not exist" containerID="91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d"
Sep 17 14:40:19.013985 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.013923    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d"} err="failed to get container status \"91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d\": rpc error: code = NotFound desc = could not find container \"91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d\": container with ID starting with 91c3aab334ca6e33d42bee287189718ec797091695c750a18e3fb96816e15f7d not found: ID does not exist"
Sep 17 14:40:19.026406 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.026383    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"]
Sep 17 14:40:19.031588 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.031565    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3da4be/rate-test-model-3da4be-kserve-6df88cf9b7-2mlv4"]
Sep 17 14:40:19.094090 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.094055    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz" podUID="cae8c7b4-be75-496d-bc96-38981558ace0" containerName="main" containerID="cri-o://31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193" gracePeriod=45
Sep 17 14:40:19.382315 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.382290    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:40:19.486202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-dshm\") pod \"cae8c7b4-be75-496d-bc96-38981558ace0\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") "
Sep 17 14:40:19.486435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486220    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-tmp-dir\") pod \"cae8c7b4-be75-496d-bc96-38981558ace0\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") "
Sep 17 14:40:19.486435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486287    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-model-cache\") pod \"cae8c7b4-be75-496d-bc96-38981558ace0\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") "
Sep 17 14:40:19.486435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486336    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phvjp\" (UniqueName: \"kubernetes.io/projected/cae8c7b4-be75-496d-bc96-38981558ace0-kube-api-access-phvjp\") pod \"cae8c7b4-be75-496d-bc96-38981558ace0\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") "
Sep 17 14:40:19.486435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486368    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae8c7b4-be75-496d-bc96-38981558ace0-tls-certs\") pod \"cae8c7b4-be75-496d-bc96-38981558ace0\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") "
Sep 17 14:40:19.486435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486435    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-home\") pod \"cae8c7b4-be75-496d-bc96-38981558ace0\" (UID: \"cae8c7b4-be75-496d-bc96-38981558ace0\") "
Sep 17 14:40:19.486709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486522    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cae8c7b4-be75-496d-bc96-38981558ace0" (UID: "cae8c7b4-be75-496d-bc96-38981558ace0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:40:19.486709 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486535    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-model-cache" (OuterVolumeSpecName: "model-cache") pod "cae8c7b4-be75-496d-bc96-38981558ace0" (UID: "cae8c7b4-be75-496d-bc96-38981558ace0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:40:19.486817 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486701    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-home" (OuterVolumeSpecName: "home") pod "cae8c7b4-be75-496d-bc96-38981558ace0" (UID: "cae8c7b4-be75-496d-bc96-38981558ace0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:40:19.486959 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486936    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:19.487029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486962    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:19.487029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.486975    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:19.488494 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.488467    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-dshm" (OuterVolumeSpecName: "dshm") pod "cae8c7b4-be75-496d-bc96-38981558ace0" (UID: "cae8c7b4-be75-496d-bc96-38981558ace0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:40:19.488585 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.488553    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae8c7b4-be75-496d-bc96-38981558ace0-kube-api-access-phvjp" (OuterVolumeSpecName: "kube-api-access-phvjp") pod "cae8c7b4-be75-496d-bc96-38981558ace0" (UID: "cae8c7b4-be75-496d-bc96-38981558ace0"). InnerVolumeSpecName "kube-api-access-phvjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:40:19.488585 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.488567    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae8c7b4-be75-496d-bc96-38981558ace0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cae8c7b4-be75-496d-bc96-38981558ace0" (UID: "cae8c7b4-be75-496d-bc96-38981558ace0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:40:19.520107 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.520041    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba929e2b-6263-4a5f-8c60-db8cfa94222f" path="/var/lib/kubelet/pods/ba929e2b-6263-4a5f-8c60-db8cfa94222f/volumes"
Sep 17 14:40:19.587752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.587719    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phvjp\" (UniqueName: \"kubernetes.io/projected/cae8c7b4-be75-496d-bc96-38981558ace0-kube-api-access-phvjp\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:19.587752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.587747    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cae8c7b4-be75-496d-bc96-38981558ace0-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:19.587752 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:19.587758    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cae8c7b4-be75-496d-bc96-38981558ace0-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:40:20.010971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.010932    2565 generic.go:356] "Generic (PLEG): container finished" podID="cae8c7b4-be75-496d-bc96-38981558ace0" containerID="31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193" exitCode=0
Sep 17 14:40:20.011449 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.010978    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"
Sep 17 14:40:20.011449 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.011023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz" event={"ID":"cae8c7b4-be75-496d-bc96-38981558ace0","Type":"ContainerDied","Data":"31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193"}
Sep 17 14:40:20.011449 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.011067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz" event={"ID":"cae8c7b4-be75-496d-bc96-38981558ace0","Type":"ContainerDied","Data":"6e65f6a59e464b0d5025f61671689485a5f0ef30c082fbc0f1702a295f7d0e07"}
Sep 17 14:40:20.011449 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.011092    2565 scope.go:117] "RemoveContainer" containerID="31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193"
Sep 17 14:40:20.020017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.020001    2565 scope.go:117] "RemoveContainer" containerID="31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193"
Sep 17 14:40:20.020306 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:40:20.020287    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193\": container with ID starting with 31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193 not found: ID does not exist" containerID="31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193"
Sep 17 14:40:20.020391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.020318    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193"} err="failed to get container status \"31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193\": rpc error: code = NotFound desc = could not find container \"31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193\": container with ID starting with 31c76e7ea9dfe69fe47c6f38b69be74371c0ef27e4517b1f97569900c7946193 not found: ID does not exist"
Sep 17 14:40:20.027182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.027142    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"]
Sep 17 14:40:20.029157 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:20.029128    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-b48b7c/rate-test-model-b48b7c-kserve-86bd48b59-nqqwz"]
Sep 17 14:40:21.519717 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:21.519685    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae8c7b4-be75-496d-bc96-38981558ace0" path="/var/lib/kubelet/pods/cae8c7b4-be75-496d-bc96-38981558ace0/volumes"
Sep 17 14:40:21.957307 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:21.957276    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:40:21.957504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:21.957353    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:40:22.959408 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:22.959370    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 14:40:22.959787 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:22.959473    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 14:40:24.038125 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:24.038091    2565 generic.go:356] "Generic (PLEG): container finished" podID="29c142b9-ad5b-4e45-8588-622475787825" containerID="72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9" exitCode=0
Sep 17 14:40:24.038583 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:24.038161    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" event={"ID":"29c142b9-ad5b-4e45-8588-622475787825","Type":"ContainerDied","Data":"72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9"}
Sep 17 14:40:24.038648 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:24.038604    2565 scope.go:117] "RemoveContainer" containerID="72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9"
Sep 17 14:40:24.039755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:24.039727    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:24.039869 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:24.039802    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:24.039869 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:24.039849    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:25.046136 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:25.046099    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" event={"ID":"29c142b9-ad5b-4e45-8588-622475787825","Type":"ContainerStarted","Data":"af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de"}
Sep 17 14:40:25.275952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:25.275905    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:25.276194 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:25.276167    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 17 14:40:25.276317 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:25.276223    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 17 14:40:26.276121 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:26.276080    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 17 14:40:26.276597 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:26.276152    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 17 14:40:27.276086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:27.276046    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 17 14:40:27.276426 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:27.276392    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 17 14:40:28.276377 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:28.276339    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 17 14:40:28.276575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:28.276408    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 17 14:40:29.276716 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:29.276676    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body=
Sep 17 14:40:29.277366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:29.277331    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused"
Sep 17 14:40:30.279624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:30.279589    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:30.280081 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:30.279947    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:30.280659 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:30.280641    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:40:30.515161 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:30.515121    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:31.963124 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:31.963083    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:32.756414 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:32.756369    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:33.140485 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.140451    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:33.579670 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.579632    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:33.739005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.738962    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:33.854141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.854041    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"]
Sep 17 14:40:33.854745 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.854720    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae8c7b4-be75-496d-bc96-38981558ace0" containerName="main"
Sep 17 14:40:33.854856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.854747    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae8c7b4-be75-496d-bc96-38981558ace0" containerName="main"
Sep 17 14:40:33.854856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.854761    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba929e2b-6263-4a5f-8c60-db8cfa94222f" containerName="main"
Sep 17 14:40:33.854856 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.854771    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba929e2b-6263-4a5f-8c60-db8cfa94222f" containerName="main"
Sep 17 14:40:33.855010 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.854886    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba929e2b-6263-4a5f-8c60-db8cfa94222f" containerName="main"
Sep 17 14:40:33.855010 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.854902    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae8c7b4-be75-496d-bc96-38981558ace0" containerName="main"
Sep 17 14:40:33.858378 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.858358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.861077 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.861050    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5943ae-openshift-default-dockercfg-kxv97\""
Sep 17 14:40:33.871735 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.871708    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"]
Sep 17 14:40:33.904458 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.904429    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:33.939860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.939818    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-socket\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.939860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.939865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-credential-socket\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.940089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.939940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161e5834-163d-42ea-9486-627db3031246-istio-podinfo\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.940089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.940024    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-envoy\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.940089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.940055    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161e5834-163d-42ea-9486-627db3031246-istiod-ca-cert\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.940089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.940073    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdl6g\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-kube-api-access-fdl6g\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.940223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.940121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-istio-token\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.940223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.940145    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-data\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:33.940223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:33.940161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-certs\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.040782 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.040750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-istio-token\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.040799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-data\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.040971    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-certs\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041176 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041140    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-socket\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-credential-socket\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161e5834-163d-42ea-9486-627db3031246-istio-podinfo\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-envoy\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041324    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-data\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161e5834-163d-42ea-9486-627db3031246-istiod-ca-cert\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-certs\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041411 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdl6g\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-kube-api-access-fdl6g\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041676 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041537    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-credential-socket\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.041781 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.041763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-socket\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.042126 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.042099    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161e5834-163d-42ea-9486-627db3031246-istiod-ca-cert\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.043952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.043924    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-envoy\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.044284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.044237    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161e5834-163d-42ea-9486-627db3031246-istio-podinfo\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.049633 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.049610    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-istio-token\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.056928 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.056370    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdl6g\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-kube-api-access-fdl6g\") pod \"e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") " pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.169823 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.169711    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:34.468309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.468187    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:34.477108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.477070    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"]
Sep 17 14:40:34.479394 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:40:34.479362    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod161e5834_163d_42ea_9486_627db3031246.slice/crio-0807387b6cffc4dce59be5af5c3a92766835163e92c5b3697d747b356c406d88 WatchSource:0}: Error finding container 0807387b6cffc4dce59be5af5c3a92766835163e92c5b3697d747b356c406d88: Status 404 returned error can't find the container with id 0807387b6cffc4dce59be5af5c3a92766835163e92c5b3697d747b356c406d88
Sep 17 14:40:34.481927 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.481893    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:34.482037 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.481976    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:34.482037 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.482020    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:34.650182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:34.650124    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:35.123734 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:35.123692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" event={"ID":"161e5834-163d-42ea-9486-627db3031246","Type":"ContainerStarted","Data":"2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd"}
Sep 17 14:40:35.123915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:35.123789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" event={"ID":"161e5834-163d-42ea-9486-627db3031246","Type":"ContainerStarted","Data":"0807387b6cffc4dce59be5af5c3a92766835163e92c5b3697d747b356c406d88"}
Sep 17 14:40:35.146141 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:35.146083    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" podStartSLOduration=2.146062574 podStartE2EDuration="2.146062574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:35.14510664 +0000 UTC m=+2022.078611831" watchObservedRunningTime="2026-09-17 14:40:35.146062574 +0000 UTC m=+2022.079567774"
Sep 17 14:40:35.170891 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:35.170859    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:35.176366 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:35.176336    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:36.129630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:36.129590    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:36.130539 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:36.130518    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:36.403513 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:36.403424    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:37.079393 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:37.079361    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:37.178160 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:37.178113    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:38.047530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:38.047480    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:38.834925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:38.834893    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:39.630874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:39.630839    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:39.732548 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:39.732520    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:39.930716 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:39.930634    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:40.216038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.215945    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:40.311437 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.311393    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:40.321468 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.321434    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"]
Sep 17 14:40:40.326099 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.326083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:40.329432 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.329406    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5943ae-dockercfg-fz4hn\""
Sep 17 14:40:40.329749 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.329730    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5943ae\""
Sep 17 14:40:40.342719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.342691    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"]
Sep 17 14:40:40.353441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.352746    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"]
Sep 17 14:40:40.357528 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.357501    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:40.371688 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.371660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"]
Sep 17 14:40:40.409645 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.409610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqqrd\" (UniqueName: \"kubernetes.io/projected/18aa0643-91e8-4f14-8849-d9e358f1a552-kube-api-access-cqqrd\") pod \"payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn\" (UID: \"18aa0643-91e8-4f14-8849-d9e358f1a552\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:40.409788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.409737    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18aa0643-91e8-4f14-8849-d9e358f1a552-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn\" (UID: \"18aa0643-91e8-4f14-8849-d9e358f1a552\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:40.409839 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.409794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab359c3-493c-429d-ad64-faeb264fb2ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2\" (UID: \"eab359c3-493c-429d-ad64-faeb264fb2ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:40.409839 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.409817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mv2q\" (UniqueName: \"kubernetes.io/projected/eab359c3-493c-429d-ad64-faeb264fb2ea-kube-api-access-7mv2q\") pod \"payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2\" (UID: \"eab359c3-493c-429d-ad64-faeb264fb2ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:40.511179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.511098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18aa0643-91e8-4f14-8849-d9e358f1a552-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn\" (UID: \"18aa0643-91e8-4f14-8849-d9e358f1a552\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:40.511179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.511142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab359c3-493c-429d-ad64-faeb264fb2ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2\" (UID: \"eab359c3-493c-429d-ad64-faeb264fb2ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:40.511179 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.511164    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mv2q\" (UniqueName: \"kubernetes.io/projected/eab359c3-493c-429d-ad64-faeb264fb2ea-kube-api-access-7mv2q\") pod \"payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2\" (UID: \"eab359c3-493c-429d-ad64-faeb264fb2ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:40.511444 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.511212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqqrd\" (UniqueName: \"kubernetes.io/projected/18aa0643-91e8-4f14-8849-d9e358f1a552-kube-api-access-cqqrd\") pod \"payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn\" (UID: \"18aa0643-91e8-4f14-8849-d9e358f1a552\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:40.511864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.511842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18aa0643-91e8-4f14-8849-d9e358f1a552-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn\" (UID: \"18aa0643-91e8-4f14-8849-d9e358f1a552\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:40.511935 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.511915    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab359c3-493c-429d-ad64-faeb264fb2ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2\" (UID: \"eab359c3-493c-429d-ad64-faeb264fb2ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:40.519755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.519734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mv2q\" (UniqueName: \"kubernetes.io/projected/eab359c3-493c-429d-ad64-faeb264fb2ea-kube-api-access-7mv2q\") pod \"payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2\" (UID: \"eab359c3-493c-429d-ad64-faeb264fb2ea\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:40.519869 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.519827    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqqrd\" (UniqueName: \"kubernetes.io/projected/18aa0643-91e8-4f14-8849-d9e358f1a552-kube-api-access-cqqrd\") pod \"payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn\" (UID: \"18aa0643-91e8-4f14-8849-d9e358f1a552\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:40.638122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.638092    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:40.670730 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.670697    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:40.915586 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.915558    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"]
Sep 17 14:40:40.917354 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:40:40.917326    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab359c3_493c_429d_ad64_faeb264fb2ea.slice/crio-b4917a0287edd45142888333329c6cc6dee8d94425c2527d488d4d76dc5b9bbf WatchSource:0}: Error finding container b4917a0287edd45142888333329c6cc6dee8d94425c2527d488d4d76dc5b9bbf: Status 404 returned error can't find the container with id b4917a0287edd45142888333329c6cc6dee8d94425c2527d488d4d76dc5b9bbf
Sep 17 14:40:40.961972 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:40.961945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"]
Sep 17 14:40:40.964316 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:40:40.964291    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18aa0643_91e8_4f14_8849_d9e358f1a552.slice/crio-8457f0f0b63daba099060eb63bc9572442c3b06dcedfdcb8d61f6f6cafef586b WatchSource:0}: Error finding container 8457f0f0b63daba099060eb63bc9572442c3b06dcedfdcb8d61f6f6cafef586b: Status 404 returned error can't find the container with id 8457f0f0b63daba099060eb63bc9572442c3b06dcedfdcb8d61f6f6cafef586b
Sep 17 14:40:41.161352 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:41.161314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2" event={"ID":"eab359c3-493c-429d-ad64-faeb264fb2ea","Type":"ContainerStarted","Data":"c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63"}
Sep 17 14:40:41.161542 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:41.161370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2" event={"ID":"eab359c3-493c-429d-ad64-faeb264fb2ea","Type":"ContainerStarted","Data":"b4917a0287edd45142888333329c6cc6dee8d94425c2527d488d4d76dc5b9bbf"}
Sep 17 14:40:41.161542 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:41.161401    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:41.163102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:41.163072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn" event={"ID":"18aa0643-91e8-4f14-8849-d9e358f1a552","Type":"ContainerStarted","Data":"ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1"}
Sep 17 14:40:41.163234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:41.163118    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn" event={"ID":"18aa0643-91e8-4f14-8849-d9e358f1a552","Type":"ContainerStarted","Data":"8457f0f0b63daba099060eb63bc9572442c3b06dcedfdcb8d61f6f6cafef586b"}
Sep 17 14:40:41.163234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:41.163139    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:41.178549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:41.178468    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2" podStartSLOduration=1.178453636 podStartE2EDuration="1.178453636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:41.176539924 +0000 UTC m=+2028.110045116" watchObservedRunningTime="2026-09-17 14:40:41.178453636 +0000 UTC m=+2028.111958827"
Sep 17 14:40:41.191945 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:41.191901    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn" podStartSLOduration=1.191886564 podStartE2EDuration="1.191886564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:40:41.190999422 +0000 UTC m=+2028.124504627" watchObservedRunningTime="2026-09-17 14:40:41.191886564 +0000 UTC m=+2028.125391755"
Sep 17 14:40:46.131349 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:46.131316    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:40:52.172221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:52.172176    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:40:52.173602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:52.173580    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:40:54.095702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:54.095661    2565 scope.go:117] "RemoveContainer" containerID="b3f9cd1bc8d7e335640cf52f0d7712c9c4634f414dee3b0c04f8e0d6e1a45852"
Sep 17 14:40:54.107700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:54.107677    2565 scope.go:117] "RemoveContainer" containerID="15ea0f4b2938c93e69e32be1855ecdb088d31197c2c3790e9152ae63890f9644"
Sep 17 14:40:57.264866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:57.264825    2565 generic.go:356] "Generic (PLEG): container finished" podID="161e5834-163d-42ea-9486-627db3031246" containerID="2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd" exitCode=0
Sep 17 14:40:57.265504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:57.264904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" event={"ID":"161e5834-163d-42ea-9486-627db3031246","Type":"ContainerDied","Data":"2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd"}
Sep 17 14:40:57.265504 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:57.265343    2565 scope.go:117] "RemoveContainer" containerID="2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd"
Sep 17 14:40:57.266567 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:57.266539    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:57.266644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:57.266610    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:57.266644 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:57.266640    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 17 14:40:58.276568 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:58.276531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" event={"ID":"161e5834-163d-42ea-9486-627db3031246","Type":"ContainerStarted","Data":"e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5"}
Sep 17 14:40:59.170860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:59.170824    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:40:59.171291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:59.171177    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:40:59.171495 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:40:59.171472    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 17 14:41:00.171298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:00.171226    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:41:00.171918 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:00.171324    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 17 14:41:00.604298 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:00.604247    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:01.171213 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:01.171170    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:41:01.171453 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:01.171279    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 17 14:41:02.047781 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:02.047739    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:02.146943 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:02.146908    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:02.170579 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:02.170540    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Sep 17 14:41:02.170780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:02.170623    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Sep 17 14:41:02.965038 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:02.965004    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:03.362017 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:03.361981    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:41:03.362554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:03.362530    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:41:03.363943 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:03.363919    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:41:03.704156 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:03.704068    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:04.419430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:04.419391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:05.369968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:05.369934    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:07.226531 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:07.226491    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn_18aa0643-91e8-4f14-8849-d9e358f1a552/payload-processing/0.log"
Sep 17 14:41:23.931636 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:23.931427    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"]
Sep 17 14:41:23.938170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:23.938133    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:23.943834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:23.943801    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c2db32\"/\"ipp-route-c2db32-kserve-self-signed-certs\""
Sep 17 14:41:23.944064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:23.943844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c2db32\"/\"default-dockercfg-wcmjd\""
Sep 17 14:41:23.944064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:23.943818    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c2db32\"/\"kube-root-ca.crt\""
Sep 17 14:41:23.944064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:23.943936    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c2db32\"/\"openshift-service-ca.crt\""
Sep 17 14:41:23.947805 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:23.947780    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"]
Sep 17 14:41:23.969187 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:23.969144    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:24.069362 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.069318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-home\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.069566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.069371    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-dshm\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.069566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.069476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d44953-9f59-41e3-a4d0-7709b3de53fe-tls-certs\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.069669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.069599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-tmp-dir\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.069669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.069631    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gdg9\" (UniqueName: \"kubernetes.io/projected/90d44953-9f59-41e3-a4d0-7709b3de53fe-kube-api-access-2gdg9\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.069669 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.069659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-model-cache\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171283 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171215    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d44953-9f59-41e3-a4d0-7709b3de53fe-tls-certs\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-tmp-dir\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171333    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gdg9\" (UniqueName: \"kubernetes.io/projected/90d44953-9f59-41e3-a4d0-7709b3de53fe-kube-api-access-2gdg9\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-model-cache\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-home\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-dshm\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171864 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171841    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-tmp-dir\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171932 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171900    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-model-cache\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.171969 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.171908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-home\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.173810 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.173779    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-dshm\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.173941 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.173923    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d44953-9f59-41e3-a4d0-7709b3de53fe-tls-certs\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.180109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.180078    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gdg9\" (UniqueName: \"kubernetes.io/projected/90d44953-9f59-41e3-a4d0-7709b3de53fe-kube-api-access-2gdg9\") pod \"ipp-route-c2db32-kserve-5d4c986d6c-jg684\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") " pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.249139 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.249042    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:24.528057 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:24.528030    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"]
Sep 17 14:41:24.529771 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:41:24.529737    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90d44953_9f59_41e3_a4d0_7709b3de53fe.slice/crio-07639e3334bcddd2ec06581f91216135d29c2b007e1db344d43343026404c6a2 WatchSource:0}: Error finding container 07639e3334bcddd2ec06581f91216135d29c2b007e1db344d43343026404c6a2: Status 404 returned error can't find the container with id 07639e3334bcddd2ec06581f91216135d29c2b007e1db344d43343026404c6a2
Sep 17 14:41:25.428442 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:25.428408    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684" event={"ID":"90d44953-9f59-41e3-a4d0-7709b3de53fe","Type":"ContainerStarted","Data":"fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb"}
Sep 17 14:41:25.428442 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:25.428454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684" event={"ID":"90d44953-9f59-41e3-a4d0-7709b3de53fe","Type":"ContainerStarted","Data":"07639e3334bcddd2ec06581f91216135d29c2b007e1db344d43343026404c6a2"}
Sep 17 14:41:25.451966 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:25.451910    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684" podStartSLOduration=2.4518894270000002 podStartE2EDuration="2.451889427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:41:25.449314583 +0000 UTC m=+2072.382819772" watchObservedRunningTime="2026-09-17 14:41:25.451889427 +0000 UTC m=+2072.385394624"
Sep 17 14:41:34.250205 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:34.250143    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:34.250205 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:34.250192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:34.263301 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:34.263273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:34.496938 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:34.496903    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:41:53.483045 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:53.483006    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds_4e56d6f1-2b53-4194-89cb-76dcc45dd282/payload-processing/0.log"
Sep 17 14:41:53.663996 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:53.663959    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:41:53.669860 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:53.669838    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:41:53.672145 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:53.672118    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:41:53.677333 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:53.677315    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:41:53.735114 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:53.735047    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn_18aa0643-91e8-4f14-8849-d9e358f1a552/payload-processing/0.log"
Sep 17 14:41:53.860629 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:53.860604    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn_18aa0643-91e8-4f14-8849-d9e358f1a552/payload-processing/0.log"
Sep 17 14:41:54.138846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:54.138815    2565 scope.go:117] "RemoveContainer" containerID="a0e9811618aefd44d1e6d29f93490d48b8094a26bf1f5f213479da18dd93667c"
Sep 17 14:41:54.149580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:54.149507    2565 scope.go:117] "RemoveContainer" containerID="02509ff452fbd8c00c6bd53a921d874daf8616519622ab4003bfc60914a1bbda"
Sep 17 14:41:54.157722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:54.157701    2565 scope.go:117] "RemoveContainer" containerID="36574c73ebedc5f10a7fb47298a28c6572c3f51e5cccf788579f700b7172344d"
Sep 17 14:41:54.205060 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:54.205034    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:54.551572 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:54.551463    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:54.650560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:54.650532    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:57.444724 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:57.444683    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:58.202748 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:58.202714    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:58.316997 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:58.316964    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:58.654205 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:58.654169    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:41:59.571977 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:41:59.571944    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:00.666884 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:00.666855    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:01.529817 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:01.529787    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:02.430896 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:02.430852    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:02.589859 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:02.589824    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:02.869604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:02.869559    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:02.972363 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:02.972324    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:03.074620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.074587    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:03.138203 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.138119    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"]
Sep 17 14:42:03.140950 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.140929    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:03.145061 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.145039    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8fb478\""
Sep 17 14:42:03.145196 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.145064    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8fb478-dockercfg-4whbq\""
Sep 17 14:42:03.168675 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.168637    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"]
Sep 17 14:42:03.269367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.269332    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk\" (UID: \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:03.269541 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.269494    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pf9z\" (UniqueName: \"kubernetes.io/projected/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-kube-api-access-7pf9z\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk\" (UID: \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:03.370512 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.370469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pf9z\" (UniqueName: \"kubernetes.io/projected/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-kube-api-access-7pf9z\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk\" (UID: \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:03.370706 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.370572    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk\" (UID: \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:03.371240 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.371219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk\" (UID: \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:03.385697 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.385668    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pf9z\" (UniqueName: \"kubernetes.io/projected/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-kube-api-access-7pf9z\") pod \"payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk\" (UID: \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:03.452428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.452333    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:03.751556 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:03.751472    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"]
Sep 17 14:42:03.759547 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:42:03.759518    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53da2c6_6991_4a71_95a0_a8c8c2db0b33.slice/crio-7ff43ecf639141d191f9efe815f68e48c4cc66d728fc2dfb4800fdb0cf256915 WatchSource:0}: Error finding container 7ff43ecf639141d191f9efe815f68e48c4cc66d728fc2dfb4800fdb0cf256915: Status 404 returned error can't find the container with id 7ff43ecf639141d191f9efe815f68e48c4cc66d728fc2dfb4800fdb0cf256915
Sep 17 14:42:04.655603 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:04.655561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk" event={"ID":"d53da2c6-6991-4a71-95a0-a8c8c2db0b33","Type":"ContainerStarted","Data":"56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2"}
Sep 17 14:42:04.656050 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:04.655625    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk" event={"ID":"d53da2c6-6991-4a71-95a0-a8c8c2db0b33","Type":"ContainerStarted","Data":"7ff43ecf639141d191f9efe815f68e48c4cc66d728fc2dfb4800fdb0cf256915"}
Sep 17 14:42:04.656050 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:04.655652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:04.676557 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:04.676506    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk" podStartSLOduration=1.6764896089999999 podStartE2EDuration="1.676489609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:42:04.674001441 +0000 UTC m=+2111.607506633" watchObservedRunningTime="2026-09-17 14:42:04.676489609 +0000 UTC m=+2111.609994801"
Sep 17 14:42:15.664693 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:15.664661    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:25.121550 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:25.121510    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:26.193915 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:26.193877    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:27.219101 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:27.219072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:28.176134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:28.176100    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:28.311479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:28.311446    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:28.837968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:28.837934    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:28.963011 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:28.962979    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"]
Sep 17 14:42:28.965488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:28.965468    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"
Sep 17 14:42:28.967699 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:28.967677    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:42:28.967801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:28.967677    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:42:28.976102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:28.976075    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"]
Sep 17 14:42:29.033452 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:29.033422    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e885368-065f-499d-a2ec-7d5dd89e81c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj\" (UID: \"0e885368-065f-499d-a2ec-7d5dd89e81c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"
Sep 17 14:42:29.134861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:29.134774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e885368-065f-499d-a2ec-7d5dd89e81c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj\" (UID: \"0e885368-065f-499d-a2ec-7d5dd89e81c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"
Sep 17 14:42:29.135456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:29.135437    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e885368-065f-499d-a2ec-7d5dd89e81c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj\" (UID: \"0e885368-065f-499d-a2ec-7d5dd89e81c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"
Sep 17 14:42:29.277434 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:29.277402    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"
Sep 17 14:42:29.547164 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:29.547139    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"]
Sep 17 14:42:29.595573 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:29.595549    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:29.805920 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:29.805851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj" event={"ID":"0e885368-065f-499d-a2ec-7d5dd89e81c4","Type":"ContainerStarted","Data":"6f46290bd3d67b91b3bf5f1b1e9e42df5fb9514bb28641fe774f576eeed8eed6"}
Sep 17 14:42:30.584575 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:30.584532    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:30.816269 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:30.816221    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e885368-065f-499d-a2ec-7d5dd89e81c4" containerID="4c08c977d75398fa7c0ecb4b81d608f997d72aa62cea5d32d4eb35183bb7240a" exitCode=0
Sep 17 14:42:30.816462 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:30.816288    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj" event={"ID":"0e885368-065f-499d-a2ec-7d5dd89e81c4","Type":"ContainerDied","Data":"4c08c977d75398fa7c0ecb4b81d608f997d72aa62cea5d32d4eb35183bb7240a"}
Sep 17 14:42:31.352536 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:31.352501    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:31.966300 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:31.966274    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"
Sep 17 14:42:32.062751 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:32.062718    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e885368-065f-499d-a2ec-7d5dd89e81c4-maas-api-service-ca\") pod \"0e885368-065f-499d-a2ec-7d5dd89e81c4\" (UID: \"0e885368-065f-499d-a2ec-7d5dd89e81c4\") "
Sep 17 14:42:32.063079 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:32.063056    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e885368-065f-499d-a2ec-7d5dd89e81c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e885368-065f-499d-a2ec-7d5dd89e81c4" (UID: "0e885368-065f-499d-a2ec-7d5dd89e81c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:42:32.164170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:32.164089    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e885368-065f-499d-a2ec-7d5dd89e81c4-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:42:32.515855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:32.515770    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:32.828966 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:32.828931    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"
Sep 17 14:42:32.829159 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:32.828964    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj" event={"ID":"0e885368-065f-499d-a2ec-7d5dd89e81c4","Type":"ContainerDied","Data":"6f46290bd3d67b91b3bf5f1b1e9e42df5fb9514bb28641fe774f576eeed8eed6"}
Sep 17 14:42:32.829159 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:32.829014    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f46290bd3d67b91b3bf5f1b1e9e42df5fb9514bb28641fe774f576eeed8eed6"
Sep 17 14:42:33.170680 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:33.170591    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:34.177029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:34.176998    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:39.707327 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:39.707299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:40.357462 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.357428    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:40.362863 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.362828    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"]
Sep 17 14:42:40.363072 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.363049    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk" podUID="d53da2c6-6991-4a71-95a0-a8c8c2db0b33" containerName="payload-pre-processing" containerID="cri-o://56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2" gracePeriod=30
Sep 17 14:42:40.452957 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.452927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:40.539427 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.539392    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"]
Sep 17 14:42:40.540678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.540581    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:40.542939 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.542904    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-8fb478-kk5bj"]
Sep 17 14:42:40.604374 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.604352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:40.624919 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.624822    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:40.751345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.751309    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pf9z\" (UniqueName: \"kubernetes.io/projected/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-kube-api-access-7pf9z\") pod \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\" (UID: \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\") "
Sep 17 14:42:40.751806 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.751383    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-plugins-config-volume\") pod \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\" (UID: \"d53da2c6-6991-4a71-95a0-a8c8c2db0b33\") "
Sep 17 14:42:40.752095 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.752068    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d53da2c6-6991-4a71-95a0-a8c8c2db0b33" (UID: "d53da2c6-6991-4a71-95a0-a8c8c2db0b33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:42:40.754311 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.754285    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-kube-api-access-7pf9z" (OuterVolumeSpecName: "kube-api-access-7pf9z") pod "d53da2c6-6991-4a71-95a0-a8c8c2db0b33" (UID: "d53da2c6-6991-4a71-95a0-a8c8c2db0b33"). InnerVolumeSpecName "kube-api-access-7pf9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:42:40.853033 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.853003    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pf9z\" (UniqueName: \"kubernetes.io/projected/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-kube-api-access-7pf9z\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:42:40.853033 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.853031    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d53da2c6-6991-4a71-95a0-a8c8c2db0b33-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:42:40.874793 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.874761    2565 generic.go:356] "Generic (PLEG): container finished" podID="d53da2c6-6991-4a71-95a0-a8c8c2db0b33" containerID="56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2" exitCode=0
Sep 17 14:42:40.874897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.874813    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"
Sep 17 14:42:40.874897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.874851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk" event={"ID":"d53da2c6-6991-4a71-95a0-a8c8c2db0b33","Type":"ContainerDied","Data":"56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2"}
Sep 17 14:42:40.874998 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.874906    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk" event={"ID":"d53da2c6-6991-4a71-95a0-a8c8c2db0b33","Type":"ContainerDied","Data":"7ff43ecf639141d191f9efe815f68e48c4cc66d728fc2dfb4800fdb0cf256915"}
Sep 17 14:42:40.874998 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.874931    2565 scope.go:117] "RemoveContainer" containerID="56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2"
Sep 17 14:42:40.884765 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.884749    2565 scope.go:117] "RemoveContainer" containerID="56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2"
Sep 17 14:42:40.885036 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:42:40.885016    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2\": container with ID starting with 56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2 not found: ID does not exist" containerID="56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2"
Sep 17 14:42:40.885101 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.885044    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2"} err="failed to get container status \"56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2\": rpc error: code = NotFound desc = could not find container \"56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2\": container with ID starting with 56c2aa74e6fdc707eb963ebd6f9a0d7877bc494db96b916872c3e1b65da1fce2 not found: ID does not exist"
Sep 17 14:42:40.896668 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.896643    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"]
Sep 17 14:42:40.904345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.904325    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8fb478-7896f9c97d-q99mk"]
Sep 17 14:42:40.934944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:40.934923    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:41.045935 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:41.045883    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:41.521552 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:41.521509    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e885368-065f-499d-a2ec-7d5dd89e81c4" path="/var/lib/kubelet/pods/0e885368-065f-499d-a2ec-7d5dd89e81c4/volumes"
Sep 17 14:42:41.526520 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:41.522686    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d53da2c6-6991-4a71-95a0-a8c8c2db0b33" path="/var/lib/kubelet/pods/d53da2c6-6991-4a71-95a0-a8c8c2db0b33/volumes"
Sep 17 14:42:41.764654 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:41.764621    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:42.637037 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:42.636994    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:43.641307 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:43.641272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:44.289574 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:44.289542    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:46.938562 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:46.938475    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:47.040510 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:47.040472    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:47.212248 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:47.212163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:51.167601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.167568    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"]
Sep 17 14:42:51.168153 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.168133    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d53da2c6-6991-4a71-95a0-a8c8c2db0b33" containerName="payload-pre-processing"
Sep 17 14:42:51.168201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.168156    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d53da2c6-6991-4a71-95a0-a8c8c2db0b33" containerName="payload-pre-processing"
Sep 17 14:42:51.168201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.168167    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e885368-065f-499d-a2ec-7d5dd89e81c4" containerName="revoke-keys"
Sep 17 14:42:51.168201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.168173    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e885368-065f-499d-a2ec-7d5dd89e81c4" containerName="revoke-keys"
Sep 17 14:42:51.168326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.168237    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e885368-065f-499d-a2ec-7d5dd89e81c4" containerName="revoke-keys"
Sep 17 14:42:51.168326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.168248    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d53da2c6-6991-4a71-95a0-a8c8c2db0b33" containerName="payload-pre-processing"
Sep 17 14:42:51.172772 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.172753    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"
Sep 17 14:42:51.175174 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.175151    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:42:51.175347 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.175280    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:42:51.179285 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.179241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"]
Sep 17 14:42:51.261064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.261034    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e444895-68db-4ae1-a9cf-048a22917c2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr\" (UID: \"3e444895-68db-4ae1-a9cf-048a22917c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"
Sep 17 14:42:51.362117 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.362077    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e444895-68db-4ae1-a9cf-048a22917c2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr\" (UID: \"3e444895-68db-4ae1-a9cf-048a22917c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"
Sep 17 14:42:51.362744 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.362722    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e444895-68db-4ae1-a9cf-048a22917c2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr\" (UID: \"3e444895-68db-4ae1-a9cf-048a22917c2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"
Sep 17 14:42:51.485763 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.485680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"
Sep 17 14:42:51.522062 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.522029    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:51.763610 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.763581    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"]
Sep 17 14:42:51.765037 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:42:51.765010    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e444895_68db_4ae1_a9cf_048a22917c2a.slice/crio-cb5a9b60c533bae9d1385de4e8a7ea67d0240b18a6ca058fca291018b9b1c5b9 WatchSource:0}: Error finding container cb5a9b60c533bae9d1385de4e8a7ea67d0240b18a6ca058fca291018b9b1c5b9: Status 404 returned error can't find the container with id cb5a9b60c533bae9d1385de4e8a7ea67d0240b18a6ca058fca291018b9b1c5b9
Sep 17 14:42:51.942004 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:51.941969    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr" event={"ID":"3e444895-68db-4ae1-a9cf-048a22917c2a","Type":"ContainerStarted","Data":"cb5a9b60c533bae9d1385de4e8a7ea67d0240b18a6ca058fca291018b9b1c5b9"}
Sep 17 14:42:52.353243 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:52.353204    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:52.949822 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:52.949730    2565 generic.go:356] "Generic (PLEG): container finished" podID="3e444895-68db-4ae1-a9cf-048a22917c2a" containerID="bae963c4733027de34eb3f74a8959d65d0cd70a1ae6ae04efd7afd19d491a11a" exitCode=0
Sep 17 14:42:52.950006 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:52.949814    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr" event={"ID":"3e444895-68db-4ae1-a9cf-048a22917c2a","Type":"ContainerDied","Data":"bae963c4733027de34eb3f74a8959d65d0cd70a1ae6ae04efd7afd19d491a11a"}
Sep 17 14:42:53.345602 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:53.345572    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:54.101649 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.101621    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"
Sep 17 14:42:54.138224 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.138188    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:54.184952 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.184919    2565 scope.go:117] "RemoveContainer" containerID="f5f3056fdb8fbd33ecce54902b7eb79ad433856023ddf532a17733bf039eebe8"
Sep 17 14:42:54.193351 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.193326    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e444895-68db-4ae1-a9cf-048a22917c2a-maas-api-service-ca\") pod \"3e444895-68db-4ae1-a9cf-048a22917c2a\" (UID: \"3e444895-68db-4ae1-a9cf-048a22917c2a\") "
Sep 17 14:42:54.193653 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.193630    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e444895-68db-4ae1-a9cf-048a22917c2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e444895-68db-4ae1-a9cf-048a22917c2a" (UID: "3e444895-68db-4ae1-a9cf-048a22917c2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:42:54.194025 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.194008    2565 scope.go:117] "RemoveContainer" containerID="cdf1dbba26b8cb27f94f977a43737f2d45fa58594433ab3b2b072348c2bff65b"
Sep 17 14:42:54.295001 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.294971    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e444895-68db-4ae1-a9cf-048a22917c2a-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:42:54.963021 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.962985    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr" event={"ID":"3e444895-68db-4ae1-a9cf-048a22917c2a","Type":"ContainerDied","Data":"cb5a9b60c533bae9d1385de4e8a7ea67d0240b18a6ca058fca291018b9b1c5b9"}
Sep 17 14:42:54.963021 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.963022    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"
Sep 17 14:42:54.963246 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:54.963038    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb5a9b60c533bae9d1385de4e8a7ea67d0240b18a6ca058fca291018b9b1c5b9"
Sep 17 14:42:55.154614 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:55.154552    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:56.181212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:56.181177    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:56.817358 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:56.817312    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:42:56.893734 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:42:56.893703    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:07.175894 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.175856    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:07.199615 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.199583    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"]
Sep 17 14:43:07.199844 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.199820    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds" podUID="4e56d6f1-2b53-4194-89cb-76dcc45dd282" containerName="payload-processing" containerID="cri-o://752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f" gracePeriod=30
Sep 17 14:43:07.207843 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.207813    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"]
Sep 17 14:43:07.208092 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.208063    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8" podUID="88f7d27c-3414-4a45-8773-1a9482ea3007" containerName="payload-pre-processing" containerID="cri-o://03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db" gracePeriod=30
Sep 17 14:43:07.242029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.241993    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:07.313094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.313043    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:07.354173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.354132    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"]
Sep 17 14:43:07.357986 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.357957    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c2db32-r4llr"]
Sep 17 14:43:07.374389 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.374363    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:07.482750 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.482724    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:43:07.503819 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.503792    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:43:07.533310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.533272    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e444895-68db-4ae1-a9cf-048a22917c2a" path="/var/lib/kubelet/pods/3e444895-68db-4ae1-a9cf-048a22917c2a/volumes"
Sep 17 14:43:07.631897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.631866    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f7d27c-3414-4a45-8773-1a9482ea3007-plugins-config-volume\") pod \"88f7d27c-3414-4a45-8773-1a9482ea3007\" (UID: \"88f7d27c-3414-4a45-8773-1a9482ea3007\") "
Sep 17 14:43:07.632053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.631963    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e56d6f1-2b53-4194-89cb-76dcc45dd282-plugins-config-volume\") pod \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\" (UID: \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\") "
Sep 17 14:43:07.632053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.632021    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lb8x\" (UniqueName: \"kubernetes.io/projected/88f7d27c-3414-4a45-8773-1a9482ea3007-kube-api-access-6lb8x\") pod \"88f7d27c-3414-4a45-8773-1a9482ea3007\" (UID: \"88f7d27c-3414-4a45-8773-1a9482ea3007\") "
Sep 17 14:43:07.632053 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.632041    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phkzh\" (UniqueName: \"kubernetes.io/projected/4e56d6f1-2b53-4194-89cb-76dcc45dd282-kube-api-access-phkzh\") pod \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\" (UID: \"4e56d6f1-2b53-4194-89cb-76dcc45dd282\") "
Sep 17 14:43:07.632359 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.632323    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88f7d27c-3414-4a45-8773-1a9482ea3007-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "88f7d27c-3414-4a45-8773-1a9482ea3007" (UID: "88f7d27c-3414-4a45-8773-1a9482ea3007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:43:07.632359 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.632346    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e56d6f1-2b53-4194-89cb-76dcc45dd282-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e56d6f1-2b53-4194-89cb-76dcc45dd282" (UID: "4e56d6f1-2b53-4194-89cb-76dcc45dd282"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:43:07.634193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.634170    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88f7d27c-3414-4a45-8773-1a9482ea3007-kube-api-access-6lb8x" (OuterVolumeSpecName: "kube-api-access-6lb8x") pod "88f7d27c-3414-4a45-8773-1a9482ea3007" (UID: "88f7d27c-3414-4a45-8773-1a9482ea3007"). InnerVolumeSpecName "kube-api-access-6lb8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:43:07.634193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.634179    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e56d6f1-2b53-4194-89cb-76dcc45dd282-kube-api-access-phkzh" (OuterVolumeSpecName: "kube-api-access-phkzh") pod "4e56d6f1-2b53-4194-89cb-76dcc45dd282" (UID: "4e56d6f1-2b53-4194-89cb-76dcc45dd282"). InnerVolumeSpecName "kube-api-access-phkzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:43:07.733599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.733535    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88f7d27c-3414-4a45-8773-1a9482ea3007-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:07.733599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.733561    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e56d6f1-2b53-4194-89cb-76dcc45dd282-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:07.733599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.733570    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lb8x\" (UniqueName: \"kubernetes.io/projected/88f7d27c-3414-4a45-8773-1a9482ea3007-kube-api-access-6lb8x\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:07.733599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.733579    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phkzh\" (UniqueName: \"kubernetes.io/projected/4e56d6f1-2b53-4194-89cb-76dcc45dd282-kube-api-access-phkzh\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:07.982172 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:07.982136    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:08.032202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.032107    2565 generic.go:356] "Generic (PLEG): container finished" podID="88f7d27c-3414-4a45-8773-1a9482ea3007" containerID="03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db" exitCode=0
Sep 17 14:43:08.032202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.032160    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"
Sep 17 14:43:08.032421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.032206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8" event={"ID":"88f7d27c-3414-4a45-8773-1a9482ea3007","Type":"ContainerDied","Data":"03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db"}
Sep 17 14:43:08.032421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.032276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8" event={"ID":"88f7d27c-3414-4a45-8773-1a9482ea3007","Type":"ContainerDied","Data":"f83002941fa2f2c420660daa1bd4701ca2c9d108f45b8bcd727d8a8e115fdd36"}
Sep 17 14:43:08.032421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.032306    2565 scope.go:117] "RemoveContainer" containerID="03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db"
Sep 17 14:43:08.033692 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.033672    2565 generic.go:356] "Generic (PLEG): container finished" podID="4e56d6f1-2b53-4194-89cb-76dcc45dd282" containerID="752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f" exitCode=0
Sep 17 14:43:08.033803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.033696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds" event={"ID":"4e56d6f1-2b53-4194-89cb-76dcc45dd282","Type":"ContainerDied","Data":"752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f"}
Sep 17 14:43:08.033803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.033743    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"
Sep 17 14:43:08.033803 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.033755    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds" event={"ID":"4e56d6f1-2b53-4194-89cb-76dcc45dd282","Type":"ContainerDied","Data":"6f7fcffb918d9fe1e790dd8c282f581424645c12af4dae9e5f7396246536e2d1"}
Sep 17 14:43:08.041893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.041882    2565 scope.go:117] "RemoveContainer" containerID="03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db"
Sep 17 14:43:08.042159 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:43:08.042139    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db\": container with ID starting with 03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db not found: ID does not exist" containerID="03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db"
Sep 17 14:43:08.042267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.042165    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db"} err="failed to get container status \"03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db\": rpc error: code = NotFound desc = could not find container \"03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db\": container with ID starting with 03386dccdadb4510f6e1d0b7547febe63ee1f8267be8030d718f6f07de8ee9db not found: ID does not exist"
Sep 17 14:43:08.042267 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.042181    2565 scope.go:117] "RemoveContainer" containerID="752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f"
Sep 17 14:43:08.051365 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.051290    2565 scope.go:117] "RemoveContainer" containerID="752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f"
Sep 17 14:43:08.051618 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:43:08.051590    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f\": container with ID starting with 752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f not found: ID does not exist" containerID="752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f"
Sep 17 14:43:08.051697 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.051627    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f"} err="failed to get container status \"752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f\": rpc error: code = NotFound desc = could not find container \"752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f\": container with ID starting with 752517bf7e73d5664b2c211a99deb17a5853ee863aea1456bc71dcf2ed585b2f not found: ID does not exist"
Sep 17 14:43:08.059094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.059070    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"]
Sep 17 14:43:08.062509 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.062488    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:08.063284 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.063248    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c2db32-7cd94cf49c-p5hds"]
Sep 17 14:43:08.072736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.072715    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"]
Sep 17 14:43:08.078627 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.078604    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c2db32-6b7d868774-zzwj8"]
Sep 17 14:43:08.615122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:08.615076    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:09.321779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:09.321746    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:09.519825 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:09.519791    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e56d6f1-2b53-4194-89cb-76dcc45dd282" path="/var/lib/kubelet/pods/4e56d6f1-2b53-4194-89cb-76dcc45dd282/volumes"
Sep 17 14:43:09.520183 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:09.520171    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88f7d27c-3414-4a45-8773-1a9482ea3007" path="/var/lib/kubelet/pods/88f7d27c-3414-4a45-8773-1a9482ea3007/volumes"
Sep 17 14:43:10.148361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:10.148322    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:12.702652 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:12.702605    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"]
Sep 17 14:43:12.760201 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:12.760167    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:27.722109 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:27.722061    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684" podUID="90d44953-9f59-41e3-a4d0-7709b3de53fe" containerName="main" containerID="cri-o://fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb" gracePeriod=45
Sep 17 14:43:27.968916 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:27.968888    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:43:28.036618 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.036536    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-tmp-dir\") pod \"90d44953-9f59-41e3-a4d0-7709b3de53fe\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") "
Sep 17 14:43:28.036618 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.036578    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-dshm\") pod \"90d44953-9f59-41e3-a4d0-7709b3de53fe\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") "
Sep 17 14:43:28.036912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.036632    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-model-cache\") pod \"90d44953-9f59-41e3-a4d0-7709b3de53fe\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") "
Sep 17 14:43:28.036912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.036683    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d44953-9f59-41e3-a4d0-7709b3de53fe-tls-certs\") pod \"90d44953-9f59-41e3-a4d0-7709b3de53fe\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") "
Sep 17 14:43:28.036912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.036798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-home\") pod \"90d44953-9f59-41e3-a4d0-7709b3de53fe\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") "
Sep 17 14:43:28.036912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.036829    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gdg9\" (UniqueName: \"kubernetes.io/projected/90d44953-9f59-41e3-a4d0-7709b3de53fe-kube-api-access-2gdg9\") pod \"90d44953-9f59-41e3-a4d0-7709b3de53fe\" (UID: \"90d44953-9f59-41e3-a4d0-7709b3de53fe\") "
Sep 17 14:43:28.036912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.036892    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "90d44953-9f59-41e3-a4d0-7709b3de53fe" (UID: "90d44953-9f59-41e3-a4d0-7709b3de53fe"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:28.037188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.036985    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-model-cache" (OuterVolumeSpecName: "model-cache") pod "90d44953-9f59-41e3-a4d0-7709b3de53fe" (UID: "90d44953-9f59-41e3-a4d0-7709b3de53fe"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:28.037188 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.037067    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-home" (OuterVolumeSpecName: "home") pod "90d44953-9f59-41e3-a4d0-7709b3de53fe" (UID: "90d44953-9f59-41e3-a4d0-7709b3de53fe"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:28.037326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.037245    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:28.037326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.037284    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:28.037326 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.037297    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:28.038976 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.038946    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d44953-9f59-41e3-a4d0-7709b3de53fe-kube-api-access-2gdg9" (OuterVolumeSpecName: "kube-api-access-2gdg9") pod "90d44953-9f59-41e3-a4d0-7709b3de53fe" (UID: "90d44953-9f59-41e3-a4d0-7709b3de53fe"). InnerVolumeSpecName "kube-api-access-2gdg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:43:28.039097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.038970    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d44953-9f59-41e3-a4d0-7709b3de53fe-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "90d44953-9f59-41e3-a4d0-7709b3de53fe" (UID: "90d44953-9f59-41e3-a4d0-7709b3de53fe"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:43:28.039097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.039062    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-dshm" (OuterVolumeSpecName: "dshm") pod "90d44953-9f59-41e3-a4d0-7709b3de53fe" (UID: "90d44953-9f59-41e3-a4d0-7709b3de53fe"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:43:28.138345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.138313    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90d44953-9f59-41e3-a4d0-7709b3de53fe-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:28.138345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.138340    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90d44953-9f59-41e3-a4d0-7709b3de53fe-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:28.138345 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.138350    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gdg9\" (UniqueName: \"kubernetes.io/projected/90d44953-9f59-41e3-a4d0-7709b3de53fe-kube-api-access-2gdg9\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:28.146960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.146933    2565 generic.go:356] "Generic (PLEG): container finished" podID="90d44953-9f59-41e3-a4d0-7709b3de53fe" containerID="fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb" exitCode=0
Sep 17 14:43:28.147086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.146973    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"
Sep 17 14:43:28.147086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.147011    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684" event={"ID":"90d44953-9f59-41e3-a4d0-7709b3de53fe","Type":"ContainerDied","Data":"fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb"}
Sep 17 14:43:28.147086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.147051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684" event={"ID":"90d44953-9f59-41e3-a4d0-7709b3de53fe","Type":"ContainerDied","Data":"07639e3334bcddd2ec06581f91216135d29c2b007e1db344d43343026404c6a2"}
Sep 17 14:43:28.147086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.147068    2565 scope.go:117] "RemoveContainer" containerID="fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb"
Sep 17 14:43:28.156024 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.156008    2565 scope.go:117] "RemoveContainer" containerID="fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb"
Sep 17 14:43:28.156282 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:43:28.156266    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb\": container with ID starting with fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb not found: ID does not exist" containerID="fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb"
Sep 17 14:43:28.156339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.156290    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb"} err="failed to get container status \"fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb\": rpc error: code = NotFound desc = could not find container \"fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb\": container with ID starting with fd043af072cabe64410120a700f925d03c3ab765404828592fcbfd7de2c83ccb not found: ID does not exist"
Sep 17 14:43:28.169639 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.169617    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"]
Sep 17 14:43:28.173608 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:28.173587    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c2db32/ipp-route-c2db32-kserve-5d4c986d6c-jg684"]
Sep 17 14:43:29.524893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:29.524852    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d44953-9f59-41e3-a4d0-7709b3de53fe" path="/var/lib/kubelet/pods/90d44953-9f59-41e3-a4d0-7709b3de53fe/volumes"
Sep 17 14:43:49.163476 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:49.163437    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:50.291881 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:50.291852    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:56.219430 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.219394    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:56.289466 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.289393    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:56.471134 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.471050    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"]
Sep 17 14:43:56.471361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.471332    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy" containerID="cri-o://af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de" gracePeriod=30
Sep 17 14:43:56.495775 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.495742    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:56.866691 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.866651    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"]
Sep 17 14:43:56.867108 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867093    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e444895-68db-4ae1-a9cf-048a22917c2a" containerName="revoke-keys"
Sep 17 14:43:56.867155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867110    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e444895-68db-4ae1-a9cf-048a22917c2a" containerName="revoke-keys"
Sep 17 14:43:56.867155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867121    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e56d6f1-2b53-4194-89cb-76dcc45dd282" containerName="payload-processing"
Sep 17 14:43:56.867155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867127    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e56d6f1-2b53-4194-89cb-76dcc45dd282" containerName="payload-processing"
Sep 17 14:43:56.867155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867136    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d44953-9f59-41e3-a4d0-7709b3de53fe" containerName="main"
Sep 17 14:43:56.867155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867142    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d44953-9f59-41e3-a4d0-7709b3de53fe" containerName="main"
Sep 17 14:43:56.867155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867150    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88f7d27c-3414-4a45-8773-1a9482ea3007" containerName="payload-pre-processing"
Sep 17 14:43:56.867155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867155    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="88f7d27c-3414-4a45-8773-1a9482ea3007" containerName="payload-pre-processing"
Sep 17 14:43:56.867435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867213    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="88f7d27c-3414-4a45-8773-1a9482ea3007" containerName="payload-pre-processing"
Sep 17 14:43:56.867435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867221    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e56d6f1-2b53-4194-89cb-76dcc45dd282" containerName="payload-processing"
Sep 17 14:43:56.867435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867226    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e444895-68db-4ae1-a9cf-048a22917c2a" containerName="revoke-keys"
Sep 17 14:43:56.867435 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.867234    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d44953-9f59-41e3-a4d0-7709b3de53fe" containerName="main"
Sep 17 14:43:56.871921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.871904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"
Sep 17 14:43:56.874155 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.874135    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:43:56.874287 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.874135    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:43:56.879551 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.879377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"]
Sep 17 14:43:56.925408 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:56.925364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96289a7f-7668-4c8d-9297-83539bedb1e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc\" (UID: \"96289a7f-7668-4c8d-9297-83539bedb1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"
Sep 17 14:43:57.026394 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:57.026348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96289a7f-7668-4c8d-9297-83539bedb1e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc\" (UID: \"96289a7f-7668-4c8d-9297-83539bedb1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"
Sep 17 14:43:57.027022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:57.026999    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96289a7f-7668-4c8d-9297-83539bedb1e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc\" (UID: \"96289a7f-7668-4c8d-9297-83539bedb1e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"
Sep 17 14:43:57.116303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:57.116270    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:57.185184 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:57.185096    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"
Sep 17 14:43:57.457247 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:57.457217    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"]
Sep 17 14:43:57.459374 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:43:57.459319    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96289a7f_7668_4c8d_9297_83539bedb1e5.slice/crio-c4f1f14171757e5d94382f948915fb70075e66dfb4b0bcf257c4e0de642482bb WatchSource:0}: Error finding container c4f1f14171757e5d94382f948915fb70075e66dfb4b0bcf257c4e0de642482bb: Status 404 returned error can't find the container with id c4f1f14171757e5d94382f948915fb70075e66dfb4b0bcf257c4e0de642482bb
Sep 17 14:43:57.937719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:57.937683    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:58.317264 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:58.317231    2565 generic.go:356] "Generic (PLEG): container finished" podID="96289a7f-7668-4c8d-9297-83539bedb1e5" containerID="a25dd9d6ea21d5b89442274bb920010f006a9c690f7f269136d9449100786973" exitCode=0
Sep 17 14:43:58.317440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:58.317317    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc" event={"ID":"96289a7f-7668-4c8d-9297-83539bedb1e5","Type":"ContainerDied","Data":"a25dd9d6ea21d5b89442274bb920010f006a9c690f7f269136d9449100786973"}
Sep 17 14:43:58.317440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:58.317366    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc" event={"ID":"96289a7f-7668-4c8d-9297-83539bedb1e5","Type":"ContainerStarted","Data":"c4f1f14171757e5d94382f948915fb70075e66dfb4b0bcf257c4e0de642482bb"}
Sep 17 14:43:58.668387 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:58.668357    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:43:59.452137 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:59.452112    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"
Sep 17 14:43:59.553363 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:59.553332    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96289a7f-7668-4c8d-9297-83539bedb1e5-maas-api-service-ca\") pod \"96289a7f-7668-4c8d-9297-83539bedb1e5\" (UID: \"96289a7f-7668-4c8d-9297-83539bedb1e5\") "
Sep 17 14:43:59.553746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:59.553713    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96289a7f-7668-4c8d-9297-83539bedb1e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96289a7f-7668-4c8d-9297-83539bedb1e5" (UID: "96289a7f-7668-4c8d-9297-83539bedb1e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:43:59.553908 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:59.553821    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96289a7f-7668-4c8d-9297-83539bedb1e5-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:43:59.557885 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:43:59.557861    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:00.266060 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:00.266030    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:00.282205 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:00.282170    2565 patch_prober.go:28] interesting pod/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:44:00.282392 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:00.282238    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:44:00.328783 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:00.328729    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc" event={"ID":"96289a7f-7668-4c8d-9297-83539bedb1e5","Type":"ContainerDied","Data":"c4f1f14171757e5d94382f948915fb70075e66dfb4b0bcf257c4e0de642482bb"}
Sep 17 14:44:00.328961 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:00.328800    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4f1f14171757e5d94382f948915fb70075e66dfb4b0bcf257c4e0de642482bb"
Sep 17 14:44:00.328961 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:00.328740    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"
Sep 17 14:44:01.723811 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.723788    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:44:01.776791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.776689    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-data\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.776791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.776732    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-certs\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.776791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.776784    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-socket\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.777064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.776820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c142b9-ad5b-4e45-8588-622475787825-istio-podinfo\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.777064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.776857    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-istio-token\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.777064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.776969    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-credential-socket\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.777064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777037    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-envoy\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.777290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777058    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-data" (OuterVolumeSpecName: "istio-data") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:01.777290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777073    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj7zz\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-kube-api-access-rj7zz\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.777290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777072    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:01.777290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777104    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c142b9-ad5b-4e45-8588-622475787825-istiod-ca-cert\") pod \"29c142b9-ad5b-4e45-8588-622475787825\" (UID: \"29c142b9-ad5b-4e45-8588-622475787825\") "
Sep 17 14:44:01.777554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777415    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:01.777554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777511    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:01.777554 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777549    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:01.777703 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777677    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:01.777897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.777864    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c142b9-ad5b-4e45-8588-622475787825-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:01.779234 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.779210    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29c142b9-ad5b-4e45-8588-622475787825-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:44:01.779433 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.779413    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:01.779846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.779823    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-kube-api-access-rj7zz" (OuterVolumeSpecName: "kube-api-access-rj7zz") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "kube-api-access-rj7zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:01.780022 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.780005    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-istio-token" (OuterVolumeSpecName: "istio-token") pod "29c142b9-ad5b-4e45-8588-622475787825" (UID: "29c142b9-ad5b-4e45-8588-622475787825"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:01.878883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.878851    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:01.878883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.878878    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:01.878883 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.878888    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj7zz\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-kube-api-access-rj7zz\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:01.879112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.878897    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29c142b9-ad5b-4e45-8588-622475787825-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:01.879112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.878906    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29c142b9-ad5b-4e45-8588-622475787825-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:01.879112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.878915    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29c142b9-ad5b-4e45-8588-622475787825-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:01.879112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:01.878924    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29c142b9-ad5b-4e45-8588-622475787825-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:02.340666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.340630    2565 generic.go:356] "Generic (PLEG): container finished" podID="29c142b9-ad5b-4e45-8588-622475787825" containerID="af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de" exitCode=0
Sep 17 14:44:02.340861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.340679    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"
Sep 17 14:44:02.340861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.340714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" event={"ID":"29c142b9-ad5b-4e45-8588-622475787825","Type":"ContainerDied","Data":"af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de"}
Sep 17 14:44:02.340861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.340796    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6" event={"ID":"29c142b9-ad5b-4e45-8588-622475787825","Type":"ContainerDied","Data":"e4394d8cb146993c0d21f3a5ee573417fe3d70e489bca399e06ce22d4e54098f"}
Sep 17 14:44:02.340861 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.340819    2565 scope.go:117] "RemoveContainer" containerID="af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de"
Sep 17 14:44:02.350175 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.350156    2565 scope.go:117] "RemoveContainer" containerID="72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9"
Sep 17 14:44:02.358036 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.358018    2565 scope.go:117] "RemoveContainer" containerID="af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de"
Sep 17 14:44:02.358285 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:44:02.358265    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de\": container with ID starting with af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de not found: ID does not exist" containerID="af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de"
Sep 17 14:44:02.358342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.358292    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de"} err="failed to get container status \"af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de\": rpc error: code = NotFound desc = could not find container \"af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de\": container with ID starting with af18f4f26091103d3f929f4c79dfa5264f304fbcb580e96e0ffed596ca1105de not found: ID does not exist"
Sep 17 14:44:02.358342 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.358308    2565 scope.go:117] "RemoveContainer" containerID="72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9"
Sep 17 14:44:02.358567 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:44:02.358549    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9\": container with ID starting with 72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9 not found: ID does not exist" containerID="72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9"
Sep 17 14:44:02.358617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.358574    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9"} err="failed to get container status \"72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9\": rpc error: code = NotFound desc = could not find container \"72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9\": container with ID starting with 72043c8ea2256d0468a9899e6cc1d0b0d900256f3658f78b8790f6ba8c0bf5d9 not found: ID does not exist"
Sep 17 14:44:02.362630 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.362611    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"]
Sep 17 14:44:02.366738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:02.366717    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-c2db32-openshift-default-5d9f8564d6-mp5h6"]
Sep 17 14:44:03.519720 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:03.519682    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c142b9-ad5b-4e45-8588-622475787825" path="/var/lib/kubelet/pods/29c142b9-ad5b-4e45-8588-622475787825/volumes"
Sep 17 14:44:07.301350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.301321    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:07.323830 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.323792    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"]
Sep 17 14:44:07.324122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.324092    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn" podUID="18aa0643-91e8-4f14-8849-d9e358f1a552" containerName="payload-processing" containerID="cri-o://ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1" gracePeriod=30
Sep 17 14:44:07.331135 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.331081    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"]
Sep 17 14:44:07.331444 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.331379    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2" podUID="eab359c3-493c-429d-ad64-faeb264fb2ea" containerName="payload-pre-processing" containerID="cri-o://c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63" gracePeriod=30
Sep 17 14:44:07.401560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.401520    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:07.494091 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.494035    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"]
Sep 17 14:44:07.497647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.497117    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:07.497647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.497459    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5943ae-9dnlc"]
Sep 17 14:44:07.520685 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.520655    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96289a7f-7668-4c8d-9297-83539bedb1e5" path="/var/lib/kubelet/pods/96289a7f-7668-4c8d-9297-83539bedb1e5/volumes"
Sep 17 14:44:07.598791 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.598768    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:07.614795 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.614772    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:44:07.617646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.617623    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:44:07.740353 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.740316    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqqrd\" (UniqueName: \"kubernetes.io/projected/18aa0643-91e8-4f14-8849-d9e358f1a552-kube-api-access-cqqrd\") pod \"18aa0643-91e8-4f14-8849-d9e358f1a552\" (UID: \"18aa0643-91e8-4f14-8849-d9e358f1a552\") "
Sep 17 14:44:07.740516 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.740372    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mv2q\" (UniqueName: \"kubernetes.io/projected/eab359c3-493c-429d-ad64-faeb264fb2ea-kube-api-access-7mv2q\") pod \"eab359c3-493c-429d-ad64-faeb264fb2ea\" (UID: \"eab359c3-493c-429d-ad64-faeb264fb2ea\") "
Sep 17 14:44:07.740516 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.740493    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18aa0643-91e8-4f14-8849-d9e358f1a552-plugins-config-volume\") pod \"18aa0643-91e8-4f14-8849-d9e358f1a552\" (UID: \"18aa0643-91e8-4f14-8849-d9e358f1a552\") "
Sep 17 14:44:07.740647 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.740619    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab359c3-493c-429d-ad64-faeb264fb2ea-plugins-config-volume\") pod \"eab359c3-493c-429d-ad64-faeb264fb2ea\" (UID: \"eab359c3-493c-429d-ad64-faeb264fb2ea\") "
Sep 17 14:44:07.740926 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.740901    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18aa0643-91e8-4f14-8849-d9e358f1a552-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18aa0643-91e8-4f14-8849-d9e358f1a552" (UID: "18aa0643-91e8-4f14-8849-d9e358f1a552"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:07.741039 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.741015    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab359c3-493c-429d-ad64-faeb264fb2ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eab359c3-493c-429d-ad64-faeb264fb2ea" (UID: "eab359c3-493c-429d-ad64-faeb264fb2ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:07.741131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.741033    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18aa0643-91e8-4f14-8849-d9e358f1a552-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:07.742702 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.742682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab359c3-493c-429d-ad64-faeb264fb2ea-kube-api-access-7mv2q" (OuterVolumeSpecName: "kube-api-access-7mv2q") pod "eab359c3-493c-429d-ad64-faeb264fb2ea" (UID: "eab359c3-493c-429d-ad64-faeb264fb2ea"). InnerVolumeSpecName "kube-api-access-7mv2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:07.742753 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.742709    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18aa0643-91e8-4f14-8849-d9e358f1a552-kube-api-access-cqqrd" (OuterVolumeSpecName: "kube-api-access-cqqrd") pod "18aa0643-91e8-4f14-8849-d9e358f1a552" (UID: "18aa0643-91e8-4f14-8849-d9e358f1a552"). InnerVolumeSpecName "kube-api-access-cqqrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:07.842238 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.842161    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab359c3-493c-429d-ad64-faeb264fb2ea-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:07.842238 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.842188    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqqrd\" (UniqueName: \"kubernetes.io/projected/18aa0643-91e8-4f14-8849-d9e358f1a552-kube-api-access-cqqrd\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:07.842238 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.842199    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mv2q\" (UniqueName: \"kubernetes.io/projected/eab359c3-493c-429d-ad64-faeb264fb2ea-kube-api-access-7mv2q\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:07.868964 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:07.868933    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:08.375245 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.375214    2565 generic.go:356] "Generic (PLEG): container finished" podID="eab359c3-493c-429d-ad64-faeb264fb2ea" containerID="c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63" exitCode=0
Sep 17 14:44:08.375684 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.375279    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"
Sep 17 14:44:08.375684 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.375304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2" event={"ID":"eab359c3-493c-429d-ad64-faeb264fb2ea","Type":"ContainerDied","Data":"c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63"}
Sep 17 14:44:08.375684 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.375367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2" event={"ID":"eab359c3-493c-429d-ad64-faeb264fb2ea","Type":"ContainerDied","Data":"b4917a0287edd45142888333329c6cc6dee8d94425c2527d488d4d76dc5b9bbf"}
Sep 17 14:44:08.375684 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.375396    2565 scope.go:117] "RemoveContainer" containerID="c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63"
Sep 17 14:44:08.376779 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.376704    2565 generic.go:356] "Generic (PLEG): container finished" podID="18aa0643-91e8-4f14-8849-d9e358f1a552" containerID="ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1" exitCode=0
Sep 17 14:44:08.376852 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.376784    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"
Sep 17 14:44:08.376852 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.376799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn" event={"ID":"18aa0643-91e8-4f14-8849-d9e358f1a552","Type":"ContainerDied","Data":"ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1"}
Sep 17 14:44:08.376920 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.376849    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn" event={"ID":"18aa0643-91e8-4f14-8849-d9e358f1a552","Type":"ContainerDied","Data":"8457f0f0b63daba099060eb63bc9572442c3b06dcedfdcb8d61f6f6cafef586b"}
Sep 17 14:44:08.386060 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.386043    2565 scope.go:117] "RemoveContainer" containerID="c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63"
Sep 17 14:44:08.386332 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:44:08.386310    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63\": container with ID starting with c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63 not found: ID does not exist" containerID="c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63"
Sep 17 14:44:08.386437 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.386337    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63"} err="failed to get container status \"c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63\": rpc error: code = NotFound desc = could not find container \"c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63\": container with ID starting with c68e9feda1e30fc13dc75df693ef01101f46e3345d2b321bc8c0481530309c63 not found: ID does not exist"
Sep 17 14:44:08.386437 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.386353    2565 scope.go:117] "RemoveContainer" containerID="ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1"
Sep 17 14:44:08.394182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.394167    2565 scope.go:117] "RemoveContainer" containerID="ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1"
Sep 17 14:44:08.394471 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:44:08.394454    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1\": container with ID starting with ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1 not found: ID does not exist" containerID="ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1"
Sep 17 14:44:08.394525 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.394477    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1"} err="failed to get container status \"ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1\": rpc error: code = NotFound desc = could not find container \"ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1\": container with ID starting with ac2027c4970753b784018b66a9cae7df74682cbb8987a0d56efa449729045fe1 not found: ID does not exist"
Sep 17 14:44:08.431906 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.431868    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"]
Sep 17 14:44:08.436291 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.436241    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5943ae-5b54b9859c-dlqwn"]
Sep 17 14:44:08.454769 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.454737    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"]
Sep 17 14:44:08.460471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.460446    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5943ae-fc5f65f66-l4st2"]
Sep 17 14:44:08.529385 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:08.529353    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:09.264454 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:09.264415    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:09.521230 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:09.521129    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18aa0643-91e8-4f14-8849-d9e358f1a552" path="/var/lib/kubelet/pods/18aa0643-91e8-4f14-8849-d9e358f1a552/volumes"
Sep 17 14:44:09.521755 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:09.521731    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eab359c3-493c-429d-ad64-faeb264fb2ea" path="/var/lib/kubelet/pods/eab359c3-493c-429d-ad64-faeb264fb2ea/volumes"
Sep 17 14:44:14.415431 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.415397    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:14.487624 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.487589    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"]
Sep 17 14:44:14.487833 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.487809    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy" containerID="cri-o://e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5" gracePeriod=30
Sep 17 14:44:14.495862 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.495834    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:14.582989 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.582951    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:14.860920 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.860879    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"]
Sep 17 14:44:14.861356 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861342    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy"
Sep 17 14:44:14.861409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861358    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy"
Sep 17 14:44:14.861409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861374    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eab359c3-493c-429d-ad64-faeb264fb2ea" containerName="payload-pre-processing"
Sep 17 14:44:14.861409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861379    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eab359c3-493c-429d-ad64-faeb264fb2ea" containerName="payload-pre-processing"
Sep 17 14:44:14.861409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861392    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy"
Sep 17 14:44:14.861409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861399    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy"
Sep 17 14:44:14.861409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861409    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96289a7f-7668-4c8d-9297-83539bedb1e5" containerName="revoke-keys"
Sep 17 14:44:14.861617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861415    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="96289a7f-7668-4c8d-9297-83539bedb1e5" containerName="revoke-keys"
Sep 17 14:44:14.861617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861424    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18aa0643-91e8-4f14-8849-d9e358f1a552" containerName="payload-processing"
Sep 17 14:44:14.861617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861430    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="18aa0643-91e8-4f14-8849-d9e358f1a552" containerName="payload-processing"
Sep 17 14:44:14.861617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861491    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy"
Sep 17 14:44:14.861617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861500    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29c142b9-ad5b-4e45-8588-622475787825" containerName="istio-proxy"
Sep 17 14:44:14.861617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861509    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="18aa0643-91e8-4f14-8849-d9e358f1a552" containerName="payload-processing"
Sep 17 14:44:14.861617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861514    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="96289a7f-7668-4c8d-9297-83539bedb1e5" containerName="revoke-keys"
Sep 17 14:44:14.861617 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.861520    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eab359c3-493c-429d-ad64-faeb264fb2ea" containerName="payload-pre-processing"
Sep 17 14:44:14.866719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.866696    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"
Sep 17 14:44:14.870870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.870842    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:44:14.870870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.870842    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:44:14.875642 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:14.875613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"]
Sep 17 14:44:15.014339 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:15.014301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8a6482-5184-40c7-b97c-72af59caddd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4\" (UID: \"cb8a6482-5184-40c7-b97c-72af59caddd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"
Sep 17 14:44:15.115911 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:15.115822    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8a6482-5184-40c7-b97c-72af59caddd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4\" (UID: \"cb8a6482-5184-40c7-b97c-72af59caddd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"
Sep 17 14:44:15.116549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:15.116529    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8a6482-5184-40c7-b97c-72af59caddd7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4\" (UID: \"cb8a6482-5184-40c7-b97c-72af59caddd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"
Sep 17 14:44:15.171549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:15.171503    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:15.178926 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:15.178902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"
Sep 17 14:44:15.358189 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:15.358158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:15.454857 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:15.454820    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"]
Sep 17 14:44:15.455853 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:44:15.455818    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb8a6482_5184_40c7_b97c_72af59caddd7.slice/crio-536b7fc0ff46a1c668f8786b588fddc13377e86b908b29ea2cb198af4414605c WatchSource:0}: Error finding container 536b7fc0ff46a1c668f8786b588fddc13377e86b908b29ea2cb198af4414605c: Status 404 returned error can't find the container with id 536b7fc0ff46a1c668f8786b588fddc13377e86b908b29ea2cb198af4414605c
Sep 17 14:44:16.099472 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:16.099383    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:16.432168 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:16.432082    2565 generic.go:356] "Generic (PLEG): container finished" podID="cb8a6482-5184-40c7-b97c-72af59caddd7" containerID="4b2d3d78f3f43d7fbaee4129c9cb0e8fcdb02c0e400609770e290fdad7c5132f" exitCode=0
Sep 17 14:44:16.432168 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:16.432152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4" event={"ID":"cb8a6482-5184-40c7-b97c-72af59caddd7","Type":"ContainerDied","Data":"4b2d3d78f3f43d7fbaee4129c9cb0e8fcdb02c0e400609770e290fdad7c5132f"}
Sep 17 14:44:16.432373 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:16.432210    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4" event={"ID":"cb8a6482-5184-40c7-b97c-72af59caddd7","Type":"ContainerStarted","Data":"536b7fc0ff46a1c668f8786b588fddc13377e86b908b29ea2cb198af4414605c"}
Sep 17 14:44:16.820463 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:16.820391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:17.573528 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:17.573496    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"
Sep 17 14:44:17.645309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:17.645273    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8a6482-5184-40c7-b97c-72af59caddd7-maas-api-service-ca\") pod \"cb8a6482-5184-40c7-b97c-72af59caddd7\" (UID: \"cb8a6482-5184-40c7-b97c-72af59caddd7\") "
Sep 17 14:44:17.645620 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:17.645593    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb8a6482-5184-40c7-b97c-72af59caddd7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cb8a6482-5184-40c7-b97c-72af59caddd7" (UID: "cb8a6482-5184-40c7-b97c-72af59caddd7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:17.746246 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:17.746165    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb8a6482-5184-40c7-b97c-72af59caddd7-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:17.746921 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:17.746892    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:18.365754 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:18.365706    2565 patch_prober.go:28] interesting pod/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 14:44:18.366146 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:18.365779    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 14:44:18.443710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:18.443676    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4" event={"ID":"cb8a6482-5184-40c7-b97c-72af59caddd7","Type":"ContainerDied","Data":"536b7fc0ff46a1c668f8786b588fddc13377e86b908b29ea2cb198af4414605c"}
Sep 17 14:44:18.443870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:18.443762    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="536b7fc0ff46a1c668f8786b588fddc13377e86b908b29ea2cb198af4414605c"
Sep 17 14:44:18.443870 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:18.443685    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"
Sep 17 14:44:19.734788 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.734763    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:44:19.867692 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.867668    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161e5834-163d-42ea-9486-627db3031246-istio-podinfo\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.867834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.867701    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-envoy\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.867834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.867725    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-credential-socket\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.867834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.867749    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-istio-token\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.867834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.867786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-socket\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.867834 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.867800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdl6g\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-kube-api-access-fdl6g\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.868101 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.867841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-data\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.868101 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.867888    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-certs\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.868101 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868049    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:19.868290 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161e5834-163d-42ea-9486-627db3031246-istiod-ca-cert\") pod \"161e5834-163d-42ea-9486-627db3031246\" (UID: \"161e5834-163d-42ea-9486-627db3031246\") "
Sep 17 14:44:19.868360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868123    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:19.868360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868137    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:19.868360 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868283    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-data" (OuterVolumeSpecName: "istio-data") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:19.868515 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868431    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161e5834-163d-42ea-9486-627db3031246-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:19.868569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868537    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:19.868569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868556    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161e5834-163d-42ea-9486-627db3031246-istiod-ca-cert\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:19.868569 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868567    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-credential-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:19.868738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868576    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-workload-socket\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:19.868738 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.868585    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-data\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:19.870005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.869969    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/161e5834-163d-42ea-9486-627db3031246-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 14:44:19.870094 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.870009    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-kube-api-access-fdl6g" (OuterVolumeSpecName: "kube-api-access-fdl6g") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "kube-api-access-fdl6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:19.870231 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.870208    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-istio-token" (OuterVolumeSpecName: "istio-token") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:19.870341 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.870309    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "161e5834-163d-42ea-9486-627db3031246" (UID: "161e5834-163d-42ea-9486-627db3031246"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:44:19.969407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.969376    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-istio-token\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:19.969407 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.969403    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdl6g\" (UniqueName: \"kubernetes.io/projected/161e5834-163d-42ea-9486-627db3031246-kube-api-access-fdl6g\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:19.969589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.969420    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161e5834-163d-42ea-9486-627db3031246-istio-podinfo\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:19.969589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:19.969433    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161e5834-163d-42ea-9486-627db3031246-istio-envoy\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:20.456721 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.456689    2565 generic.go:356] "Generic (PLEG): container finished" podID="161e5834-163d-42ea-9486-627db3031246" containerID="e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5" exitCode=0
Sep 17 14:44:20.456994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.456743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" event={"ID":"161e5834-163d-42ea-9486-627db3031246","Type":"ContainerDied","Data":"e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5"}
Sep 17 14:44:20.456994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.456808    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx" event={"ID":"161e5834-163d-42ea-9486-627db3031246","Type":"ContainerDied","Data":"0807387b6cffc4dce59be5af5c3a92766835163e92c5b3697d747b356c406d88"}
Sep 17 14:44:20.456994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.456746    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"
Sep 17 14:44:20.456994 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.456842    2565 scope.go:117] "RemoveContainer" containerID="e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5"
Sep 17 14:44:20.466880 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.466863    2565 scope.go:117] "RemoveContainer" containerID="2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd"
Sep 17 14:44:20.474725 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.474704    2565 scope.go:117] "RemoveContainer" containerID="e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5"
Sep 17 14:44:20.474971 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:44:20.474948    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5\": container with ID starting with e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5 not found: ID does not exist" containerID="e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5"
Sep 17 14:44:20.475031 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.474978    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5"} err="failed to get container status \"e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5\": rpc error: code = NotFound desc = could not find container \"e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5\": container with ID starting with e327e9732625ad01cbd1c56e490e80a65e67a343dd5a8d39be559f45178bb3a5 not found: ID does not exist"
Sep 17 14:44:20.475031 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.474993    2565 scope.go:117] "RemoveContainer" containerID="2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd"
Sep 17 14:44:20.475229 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:44:20.475211    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd\": container with ID starting with 2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd not found: ID does not exist" containerID="2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd"
Sep 17 14:44:20.475319 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.475235    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd"} err="failed to get container status \"2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd\": rpc error: code = NotFound desc = could not find container \"2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd\": container with ID starting with 2905f710b95a4c9669e9f9cde2be2a22c4a278dc1947f336a18e6b62f64c62fd not found: ID does not exist"
Sep 17 14:44:20.480415 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.480393    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"]
Sep 17 14:44:20.483131 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:20.483112    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5943ae-openshift-default-6cf8449d54-vj6cx"]
Sep 17 14:44:21.519650 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:21.519619    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="161e5834-163d-42ea-9486-627db3031246" path="/var/lib/kubelet/pods/161e5834-163d-42ea-9486-627db3031246/volumes"
Sep 17 14:44:21.860706 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:21.860675    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:25.546673 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.546631    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:25.573403 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.573361    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"]
Sep 17 14:44:25.573658 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.573632    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7" podUID="71b552e9-457f-4a56-8ce0-182c41a78d17" containerName="payload-pre-processing" containerID="cri-o://f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42" gracePeriod=30
Sep 17 14:44:25.643855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.643808    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:25.740650 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.740616    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"]
Sep 17 14:44:25.745560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.745530    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3da4be-ntjs4"]
Sep 17 14:44:25.745560 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.745547    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:25.826587 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.826556    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:25.847924 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:25.847901    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:44:26.026990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.026953    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b552e9-457f-4a56-8ce0-182c41a78d17-plugins-config-volume\") pod \"71b552e9-457f-4a56-8ce0-182c41a78d17\" (UID: \"71b552e9-457f-4a56-8ce0-182c41a78d17\") "
Sep 17 14:44:26.027170 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.027027    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xv62f\" (UniqueName: \"kubernetes.io/projected/71b552e9-457f-4a56-8ce0-182c41a78d17-kube-api-access-xv62f\") pod \"71b552e9-457f-4a56-8ce0-182c41a78d17\" (UID: \"71b552e9-457f-4a56-8ce0-182c41a78d17\") "
Sep 17 14:44:26.027417 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.027397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b552e9-457f-4a56-8ce0-182c41a78d17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "71b552e9-457f-4a56-8ce0-182c41a78d17" (UID: "71b552e9-457f-4a56-8ce0-182c41a78d17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:26.029115 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.029090    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b552e9-457f-4a56-8ce0-182c41a78d17-kube-api-access-xv62f" (OuterVolumeSpecName: "kube-api-access-xv62f") pod "71b552e9-457f-4a56-8ce0-182c41a78d17" (UID: "71b552e9-457f-4a56-8ce0-182c41a78d17"). InnerVolumeSpecName "kube-api-access-xv62f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:44:26.116925 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.116843    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:26.128204 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.128185    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b552e9-457f-4a56-8ce0-182c41a78d17-plugins-config-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:26.128294 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.128206    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xv62f\" (UniqueName: \"kubernetes.io/projected/71b552e9-457f-4a56-8ce0-182c41a78d17-kube-api-access-xv62f\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:26.216968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.216941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:26.497850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.497758    2565 generic.go:356] "Generic (PLEG): container finished" podID="71b552e9-457f-4a56-8ce0-182c41a78d17" containerID="f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42" exitCode=0
Sep 17 14:44:26.497850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.497808    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"
Sep 17 14:44:26.497850 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.497822    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7" event={"ID":"71b552e9-457f-4a56-8ce0-182c41a78d17","Type":"ContainerDied","Data":"f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42"}
Sep 17 14:44:26.498118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.497878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7" event={"ID":"71b552e9-457f-4a56-8ce0-182c41a78d17","Type":"ContainerDied","Data":"e6862057bbc70f5aa2abe546f746a9fbe921e9b67060cf58962884af4f127050"}
Sep 17 14:44:26.498118 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.497908    2565 scope.go:117] "RemoveContainer" containerID="f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42"
Sep 17 14:44:26.507802 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.507780    2565 scope.go:117] "RemoveContainer" containerID="f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42"
Sep 17 14:44:26.508059 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:44:26.508037    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42\": container with ID starting with f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42 not found: ID does not exist" containerID="f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42"
Sep 17 14:44:26.508105 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.508066    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42"} err="failed to get container status \"f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42\": rpc error: code = NotFound desc = could not find container \"f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42\": container with ID starting with f672938630aa143d6779a0a70bc1012d46886339739e5bbdbb43f6209756fb42 not found: ID does not exist"
Sep 17 14:44:26.519476 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.519453    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"]
Sep 17 14:44:26.521299 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:26.521281    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3da4be-77bcb9f4f8-6tgz7"]
Sep 17 14:44:27.032219 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:27.032183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:27.519198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:27.519164    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71b552e9-457f-4a56-8ce0-182c41a78d17" path="/var/lib/kubelet/pods/71b552e9-457f-4a56-8ce0-182c41a78d17/volumes"
Sep 17 14:44:27.519549 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:27.519536    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb8a6482-5184-40c7-b97c-72af59caddd7" path="/var/lib/kubelet/pods/cb8a6482-5184-40c7-b97c-72af59caddd7/volumes"
Sep 17 14:44:32.431499 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:32.431466    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:33.076391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076346    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"]
Sep 17 14:44:33.076848 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076827    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71b552e9-457f-4a56-8ce0-182c41a78d17" containerName="payload-pre-processing"
Sep 17 14:44:33.076960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076851    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="71b552e9-457f-4a56-8ce0-182c41a78d17" containerName="payload-pre-processing"
Sep 17 14:44:33.076960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076889    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy"
Sep 17 14:44:33.076960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076898    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy"
Sep 17 14:44:33.076960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076910    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb8a6482-5184-40c7-b97c-72af59caddd7" containerName="revoke-keys"
Sep 17 14:44:33.076960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076919    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb8a6482-5184-40c7-b97c-72af59caddd7" containerName="revoke-keys"
Sep 17 14:44:33.076960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076939    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy"
Sep 17 14:44:33.076960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.076948    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy"
Sep 17 14:44:33.077343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.077041    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy"
Sep 17 14:44:33.077343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.077055    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="161e5834-163d-42ea-9486-627db3031246" containerName="istio-proxy"
Sep 17 14:44:33.077343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.077065    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb8a6482-5184-40c7-b97c-72af59caddd7" containerName="revoke-keys"
Sep 17 14:44:33.077343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.077074    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="71b552e9-457f-4a56-8ce0-182c41a78d17" containerName="payload-pre-processing"
Sep 17 14:44:33.080464 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.080438    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"
Sep 17 14:44:33.082660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.082640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 14:44:33.082787 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.082766    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:44:33.087650 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.087624    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"]
Sep 17 14:44:33.098343 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.098319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a1883-df13-4b12-9640-a6bbb6a5d686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7\" (UID: \"c71a1883-df13-4b12-9640-a6bbb6a5d686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"
Sep 17 14:44:33.199268 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.199228    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a1883-df13-4b12-9640-a6bbb6a5d686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7\" (UID: \"c71a1883-df13-4b12-9640-a6bbb6a5d686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"
Sep 17 14:44:33.199874 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.199852    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a1883-df13-4b12-9640-a6bbb6a5d686-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7\" (UID: \"c71a1883-df13-4b12-9640-a6bbb6a5d686\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"
Sep 17 14:44:33.391852 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.391763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"
Sep 17 14:44:33.666508 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.666408    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"]
Sep 17 14:44:33.672297 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:44:33.672267    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc71a1883_df13_4b12_9640_a6bbb6a5d686.slice/crio-02e763376ddab885e79a9fb5c992ed7df9b93bcddf692ead5a871ba6d457a3a8 WatchSource:0}: Error finding container 02e763376ddab885e79a9fb5c992ed7df9b93bcddf692ead5a871ba6d457a3a8: Status 404 returned error can't find the container with id 02e763376ddab885e79a9fb5c992ed7df9b93bcddf692ead5a871ba6d457a3a8
Sep 17 14:44:33.865735 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:33.865709    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:34.490566 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:34.490528    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:34.541692 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:34.541661    2565 generic.go:356] "Generic (PLEG): container finished" podID="c71a1883-df13-4b12-9640-a6bbb6a5d686" containerID="9399d67af56bb1277e6685ef8cf98c78c0a1c59fbfe7c42dcfdfb37e87081240" exitCode=0
Sep 17 14:44:34.541873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:34.541746    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7" event={"ID":"c71a1883-df13-4b12-9640-a6bbb6a5d686","Type":"ContainerDied","Data":"9399d67af56bb1277e6685ef8cf98c78c0a1c59fbfe7c42dcfdfb37e87081240"}
Sep 17 14:44:34.541873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:34.541798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7" event={"ID":"c71a1883-df13-4b12-9640-a6bbb6a5d686","Type":"ContainerStarted","Data":"02e763376ddab885e79a9fb5c992ed7df9b93bcddf692ead5a871ba6d457a3a8"}
Sep 17 14:44:35.677710 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:35.677679    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:35.685331 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:35.685311    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"
Sep 17 14:44:35.722184 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:35.722162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a1883-df13-4b12-9640-a6bbb6a5d686-maas-api-service-ca\") pod \"c71a1883-df13-4b12-9640-a6bbb6a5d686\" (UID: \"c71a1883-df13-4b12-9640-a6bbb6a5d686\") "
Sep 17 14:44:35.722491 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:35.722466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c71a1883-df13-4b12-9640-a6bbb6a5d686-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c71a1883-df13-4b12-9640-a6bbb6a5d686" (UID: "c71a1883-df13-4b12-9640-a6bbb6a5d686"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 14:44:35.822940 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:35.822920    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c71a1883-df13-4b12-9640-a6bbb6a5d686-maas-api-service-ca\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:44:36.553722 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:36.553693    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"
Sep 17 14:44:36.553971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:36.553735    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7" event={"ID":"c71a1883-df13-4b12-9640-a6bbb6a5d686","Type":"ContainerDied","Data":"02e763376ddab885e79a9fb5c992ed7df9b93bcddf692ead5a871ba6d457a3a8"}
Sep 17 14:44:36.553971 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:36.553777    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02e763376ddab885e79a9fb5c992ed7df9b93bcddf692ead5a871ba6d457a3a8"
Sep 17 14:44:43.586347 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:43.586301    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:43.662976 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:43.662928    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:43.761309 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:43.761271    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:43.861672 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:43.861588    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:43.882413 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:43.882389    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"]
Sep 17 14:44:43.884388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:43.884366    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-b48b7c-qs8k7"]
Sep 17 14:44:44.331297 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:44.331269    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:44.434439 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:44.434407    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:45.519460 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:45.519433    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c71a1883-df13-4b12-9640-a6bbb6a5d686" path="/var/lib/kubelet/pods/c71a1883-df13-4b12-9640-a6bbb6a5d686/volumes"
Sep 17 14:44:51.553394 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:51.553359    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:51.672319 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:51.672241    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:44:51.884687 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:44:51.884654    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:45:00.242488 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.242450    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq"]
Sep 17 14:45:00.243150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.242973    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c71a1883-df13-4b12-9640-a6bbb6a5d686" containerName="revoke-keys"
Sep 17 14:45:00.243150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.242991    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c71a1883-df13-4b12-9640-a6bbb6a5d686" containerName="revoke-keys"
Sep 17 14:45:00.243150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.243063    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c71a1883-df13-4b12-9640-a6bbb6a5d686" containerName="revoke-keys"
Sep 17 14:45:00.245853 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.245837    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq"
Sep 17 14:45:00.248762 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.248735    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mm5m6\""
Sep 17 14:45:00.255785 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.255767    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq"
Sep 17 14:45:00.283440 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.283410    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq"]
Sep 17 14:45:00.532845 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.532814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq"]
Sep 17 14:45:00.533228 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:45:00.533201    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf8b174_26bb_48cc_a491_76b415bf09e7.slice/crio-e64b8b588c322b6487a11b7b62d83128fc981a37c3805974c027b910823db968 WatchSource:0}: Error finding container e64b8b588c322b6487a11b7b62d83128fc981a37c3805974c027b910823db968: Status 404 returned error can't find the container with id e64b8b588c322b6487a11b7b62d83128fc981a37c3805974c027b910823db968
Sep 17 14:45:00.676086 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:00.676050    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq" event={"ID":"acf8b174-26bb-48cc-a491-76b415bf09e7","Type":"ContainerStarted","Data":"e64b8b588c322b6487a11b7b62d83128fc981a37c3805974c027b910823db968"}
Sep 17 14:45:01.683561 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:01.683517    2565 generic.go:356] "Generic (PLEG): container finished" podID="acf8b174-26bb-48cc-a491-76b415bf09e7" containerID="24895b1f1492a259e19ff2a83a3cca620a02bdee712cafb8981adfe8b5d7cf3d" exitCode=0
Sep 17 14:45:01.683960 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:01.683602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq" event={"ID":"acf8b174-26bb-48cc-a491-76b415bf09e7","Type":"ContainerDied","Data":"24895b1f1492a259e19ff2a83a3cca620a02bdee712cafb8981adfe8b5d7cf3d"}
Sep 17 14:45:02.821223 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:02.821189    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq"
Sep 17 14:45:03.695295 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:03.695263    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq"
Sep 17 14:45:03.695295 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:03.695281    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827605-88jrq" event={"ID":"acf8b174-26bb-48cc-a491-76b415bf09e7","Type":"ContainerDied","Data":"e64b8b588c322b6487a11b7b62d83128fc981a37c3805974c027b910823db968"}
Sep 17 14:45:03.695503 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:03.695338    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64b8b588c322b6487a11b7b62d83128fc981a37c3805974c027b910823db968"
Sep 17 14:45:07.132596 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:07.132563    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:45:07.559122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:07.559006    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:45:07.633530 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:07.633500    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:45:12.506559 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:12.506515    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:45:12.608016 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:45:12.607972    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:46:20.371671 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:46:20.371634    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:46:22.255114 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:46:22.255076    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:46:42.659097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:46:42.659062    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:46:50.701232 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:46:50.701194    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:46:53.699626 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:46:53.699598    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:46:53.705232 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:46:53.705204    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:46:53.708218 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:46:53.708196    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:46:53.713663 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:46:53.713647    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:47:52.593807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:52.593773    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"]
Sep 17 14:47:52.594398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:52.594026    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerName="manager" containerID="cri-o://32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb" gracePeriod=10
Sep 17 14:47:53.436202 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.436180    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:47:53.576153 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.576125    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thqwr\" (UniqueName: \"kubernetes.io/projected/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-kube-api-access-thqwr\") pod \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\" (UID: \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\") "
Sep 17 14:47:53.576319 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.576246    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-extensions-socket-volume\") pod \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\" (UID: \"0c4317e5-a5c3-4e14-b889-0b45c3e4e082\") "
Sep 17 14:47:53.576599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.576574    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0c4317e5-a5c3-4e14-b889-0b45c3e4e082" (UID: "0c4317e5-a5c3-4e14-b889-0b45c3e4e082"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:47:53.578028 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.577998    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-kube-api-access-thqwr" (OuterVolumeSpecName: "kube-api-access-thqwr") pod "0c4317e5-a5c3-4e14-b889-0b45c3e4e082" (UID: "0c4317e5-a5c3-4e14-b889-0b45c3e4e082"). InnerVolumeSpecName "kube-api-access-thqwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:47:53.591005 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.590977    2565 generic.go:356] "Generic (PLEG): container finished" podID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerID="32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb" exitCode=0
Sep 17 14:47:53.591120 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.591020    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"
Sep 17 14:47:53.591120 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.591062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz" event={"ID":"0c4317e5-a5c3-4e14-b889-0b45c3e4e082","Type":"ContainerDied","Data":"32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb"}
Sep 17 14:47:53.591120 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.591113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz" event={"ID":"0c4317e5-a5c3-4e14-b889-0b45c3e4e082","Type":"ContainerDied","Data":"b1e37834ac2e930ceb9cb3cebadccff285371290dfc97aca1dba5878e7075af4"}
Sep 17 14:47:53.591302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.591137    2565 scope.go:117] "RemoveContainer" containerID="32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb"
Sep 17 14:47:53.599917 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.599732    2565 scope.go:117] "RemoveContainer" containerID="4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b"
Sep 17 14:47:53.607604 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.607590    2565 scope.go:117] "RemoveContainer" containerID="32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb"
Sep 17 14:47:53.607851 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:47:53.607835    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb\": container with ID starting with 32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb not found: ID does not exist" containerID="32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb"
Sep 17 14:47:53.607900 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.607858    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb"} err="failed to get container status \"32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb\": rpc error: code = NotFound desc = could not find container \"32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb\": container with ID starting with 32ab8075a4ecbdfdef0527ea6784132feeb2be9d59aca3aec79c5fbed1dc58cb not found: ID does not exist"
Sep 17 14:47:53.607900 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.607872    2565 scope.go:117] "RemoveContainer" containerID="4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b"
Sep 17 14:47:53.608110 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:47:53.608095    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b\": container with ID starting with 4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b not found: ID does not exist" containerID="4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b"
Sep 17 14:47:53.608150 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.608116    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b"} err="failed to get container status \"4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b\": rpc error: code = NotFound desc = could not find container \"4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b\": container with ID starting with 4769110c28a860cd5b91ad60118d3d8ca4ac201382dc567e298397553b38234b not found: ID does not exist"
Sep 17 14:47:53.612716 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.612680    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"]
Sep 17 14:47:53.616945 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.616923    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-lnxfz"]
Sep 17 14:47:53.677130 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.677110    2565 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-extensions-socket-volume\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:47:53.677130 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:53.677128    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thqwr\" (UniqueName: \"kubernetes.io/projected/0c4317e5-a5c3-4e14-b889-0b45c3e4e082-kube-api-access-thqwr\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:47:55.520405 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:47:55.520366    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" path="/var/lib/kubelet/pods/0c4317e5-a5c3-4e14-b889-0b45c3e4e082/volumes"
Sep 17 14:48:53.288148 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288056    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"]
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288449    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerName="manager"
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288462    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerName="manager"
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288484    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf8b174-26bb-48cc-a491-76b415bf09e7" containerName="cleanup"
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288491    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf8b174-26bb-48cc-a491-76b415bf09e7" containerName="cleanup"
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288504    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerName="manager"
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288510    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerName="manager"
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288588    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf8b174-26bb-48cc-a491-76b415bf09e7" containerName="cleanup"
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288596    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerName="manager"
Sep 17 14:48:53.288619 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.288604    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4317e5-a5c3-4e14-b889-0b45c3e4e082" containerName="manager"
Sep 17 14:48:53.291815 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.291791    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:53.295014 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.294990    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xk98g\""
Sep 17 14:48:53.309498 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.309467    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"]
Sep 17 14:48:53.469478 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.469448    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvmgf\" (UniqueName: \"kubernetes.io/projected/5a5cfc57-b949-45e9-b7c2-478b724b1aed-kube-api-access-kvmgf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks5pv\" (UID: \"5a5cfc57-b949-45e9-b7c2-478b724b1aed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:53.469601 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.469522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a5cfc57-b949-45e9-b7c2-478b724b1aed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks5pv\" (UID: \"5a5cfc57-b949-45e9-b7c2-478b724b1aed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:53.569956 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.569897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvmgf\" (UniqueName: \"kubernetes.io/projected/5a5cfc57-b949-45e9-b7c2-478b724b1aed-kube-api-access-kvmgf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks5pv\" (UID: \"5a5cfc57-b949-45e9-b7c2-478b724b1aed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:53.570063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.569954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a5cfc57-b949-45e9-b7c2-478b724b1aed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks5pv\" (UID: \"5a5cfc57-b949-45e9-b7c2-478b724b1aed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:53.570348 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.570319    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5a5cfc57-b949-45e9-b7c2-478b724b1aed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks5pv\" (UID: \"5a5cfc57-b949-45e9-b7c2-478b724b1aed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:53.578213 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.578191    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvmgf\" (UniqueName: \"kubernetes.io/projected/5a5cfc57-b949-45e9-b7c2-478b724b1aed-kube-api-access-kvmgf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ks5pv\" (UID: \"5a5cfc57-b949-45e9-b7c2-478b724b1aed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:53.604873 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.604854    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xk98g\""
Sep 17 14:48:53.612700 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.612677    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:53.889062 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.889036    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"]
Sep 17 14:48:53.890799 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:48:53.890775    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5cfc57_b949_45e9_b7c2_478b724b1aed.slice/crio-eb0426a64ac35f484a312a47e33c102e5402b3d7b47d217ce32beb68e4b65c51 WatchSource:0}: Error finding container eb0426a64ac35f484a312a47e33c102e5402b3d7b47d217ce32beb68e4b65c51: Status 404 returned error can't find the container with id eb0426a64ac35f484a312a47e33c102e5402b3d7b47d217ce32beb68e4b65c51
Sep 17 14:48:53.901686 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:53.901660    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv" event={"ID":"5a5cfc57-b949-45e9-b7c2-478b724b1aed","Type":"ContainerStarted","Data":"eb0426a64ac35f484a312a47e33c102e5402b3d7b47d217ce32beb68e4b65c51"}
Sep 17 14:48:54.308310 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:54.308207    2565 scope.go:117] "RemoveContainer" containerID="4c08c977d75398fa7c0ecb4b81d608f997d72aa62cea5d32d4eb35183bb7240a"
Sep 17 14:48:54.322550 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:54.322529    2565 scope.go:117] "RemoveContainer" containerID="bae963c4733027de34eb3f74a8959d65d0cd70a1ae6ae04efd7afd19d491a11a"
Sep 17 14:48:54.908768 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:54.908704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv" event={"ID":"5a5cfc57-b949-45e9-b7c2-478b724b1aed","Type":"ContainerStarted","Data":"abbc7cb9b00987142dfc94df152e0eef38d65dbdd8d7f43f53c0ac6731577724"}
Sep 17 14:48:54.908965 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:54.908890    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:48:54.934621 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:48:54.934579    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv" podStartSLOduration=1.9345662940000001 podStartE2EDuration="1.934566294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:48:54.932010189 +0000 UTC m=+2521.865515381" watchObservedRunningTime="2026-09-17 14:48:54.934566294 +0000 UTC m=+2521.868071488"
Sep 17 14:49:05.916660 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:05.916627    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ks5pv"
Sep 17 14:49:28.309061 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.309021    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"]
Sep 17 14:49:28.312807 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.312786    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.315165 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.315143    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c443cafd-kserve-self-signed-certs\""
Sep 17 14:49:28.319529 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.319498    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"]
Sep 17 14:49:28.375398 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.375374    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:49:28.489063 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.489030    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-home\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.489171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.489127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tmp-dir\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.489171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.489157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d7tq\" (UniqueName: \"kubernetes.io/projected/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-kube-api-access-2d7tq\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.489289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.489202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-dshm\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.489289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.489244    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-model-cache\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.489289 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.489272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tls-certs\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590242 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-home\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tmp-dir\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590328    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d7tq\" (UniqueName: \"kubernetes.io/projected/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-kube-api-access-2d7tq\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-dshm\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590428 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-model-cache\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tls-certs\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-home\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590694 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590683    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tmp-dir\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.590804 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.590723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-model-cache\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.592626 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.592596    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-dshm\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.593034 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.593012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tls-certs\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.598863 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.598840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d7tq\" (UniqueName: \"kubernetes.io/projected/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-kube-api-access-2d7tq\") pod \"e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") " pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.622704 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.622681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:28.894607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:28.894582    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"]
Sep 17 14:49:28.896350 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:49:28.896326    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a34cad8_fe71_4be5_80ab_e07dcf1568d5.slice/crio-a93932130124a96bfd4b0a35527ea45bb9510662f873d119e77a19c111decd7d WatchSource:0}: Error finding container a93932130124a96bfd4b0a35527ea45bb9510662f873d119e77a19c111decd7d: Status 404 returned error can't find the container with id a93932130124a96bfd4b0a35527ea45bb9510662f873d119e77a19c111decd7d
Sep 17 14:49:29.087540 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:29.087500    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq" event={"ID":"2a34cad8-fe71-4be5-80ab-e07dcf1568d5","Type":"ContainerStarted","Data":"706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e"}
Sep 17 14:49:29.087540 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:29.087543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq" event={"ID":"2a34cad8-fe71-4be5-80ab-e07dcf1568d5","Type":"ContainerStarted","Data":"a93932130124a96bfd4b0a35527ea45bb9510662f873d119e77a19c111decd7d"}
Sep 17 14:49:29.109717 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:29.109666    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq" podStartSLOduration=1.10965263 podStartE2EDuration="1.10965263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:49:29.106113587 +0000 UTC m=+2556.039618778" watchObservedRunningTime="2026-09-17 14:49:29.10965263 +0000 UTC m=+2556.043157823"
Sep 17 14:49:34.290269 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:34.290232    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:49:38.623358 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:38.623313    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:38.623358 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:38.623371    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:38.635866 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:38.635840    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:39.150942 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:39.150913    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:49:47.604781 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.604742    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"]
Sep 17 14:49:47.608418 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.608394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.611877 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.611856    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c443cafd-kserve-self-signed-certs\""
Sep 17 14:49:47.619456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.619430    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"]
Sep 17 14:49:47.656904 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.656874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-dshm\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.657029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.656908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-model-cache\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.657029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.656941    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db84faad-83b8-4db8-b1e5-4e4c12b79357-tls-certs\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.657029 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.657002    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjwv\" (UniqueName: \"kubernetes.io/projected/db84faad-83b8-4db8-b1e5-4e4c12b79357-kube-api-access-jfjwv\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.657169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.657066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-home\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.657169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.657106    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-tmp-dir\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.661846 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.661824    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:49:47.757990 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.757954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjwv\" (UniqueName: \"kubernetes.io/projected/db84faad-83b8-4db8-b1e5-4e4c12b79357-kube-api-access-jfjwv\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.758138 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.757999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-home\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.758138 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.758047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-tmp-dir\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.758278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.758166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-dshm\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.758278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.758194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-model-cache\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.758278 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.758239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db84faad-83b8-4db8-b1e5-4e4c12b79357-tls-certs\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.758441 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.758413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-home\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.758479 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.758441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-tmp-dir\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.758641 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.758620    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-model-cache\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.760518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.760495    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-dshm\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.760746 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.760708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db84faad-83b8-4db8-b1e5-4e4c12b79357-tls-certs\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.765867 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.765847    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjwv\" (UniqueName: \"kubernetes.io/projected/db84faad-83b8-4db8-b1e5-4e4c12b79357-kube-api-access-jfjwv\") pod \"e2e-same-id-b-c443cafd-kserve-8665496987-t25s8\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") " pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:47.920193 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:47.920127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:48.201092 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:48.201065    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"]
Sep 17 14:49:48.203009 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:49:48.202984    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddb84faad_83b8_4db8_b1e5_4e4c12b79357.slice/crio-d44f25c5073a514bb4d1b2da90e6e7e188489ead495b977f57792221ef76262d WatchSource:0}: Error finding container d44f25c5073a514bb4d1b2da90e6e7e188489ead495b977f57792221ef76262d: Status 404 returned error can't find the container with id d44f25c5073a514bb4d1b2da90e6e7e188489ead495b977f57792221ef76262d
Sep 17 14:49:49.196918 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:49.196878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8" event={"ID":"db84faad-83b8-4db8-b1e5-4e4c12b79357","Type":"ContainerStarted","Data":"f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0"}
Sep 17 14:49:49.196918 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:49.196922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8" event={"ID":"db84faad-83b8-4db8-b1e5-4e4c12b79357","Type":"ContainerStarted","Data":"d44f25c5073a514bb4d1b2da90e6e7e188489ead495b977f57792221ef76262d"}
Sep 17 14:49:49.215212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:49.215161    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8" podStartSLOduration=2.215146644 podStartE2EDuration="2.215146644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 14:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:49:49.213241631 +0000 UTC m=+2576.146746822" watchObservedRunningTime="2026-09-17 14:49:49.215146644 +0000 UTC m=+2576.148651887"
Sep 17 14:49:57.920711 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:57.920673    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:57.921112 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:57.920720    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:57.932473 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:57.932452    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:49:58.254492 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:49:58.254418    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:50:41.922302 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:41.922270    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"]
Sep 17 14:50:41.944968 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:41.944936    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:50:43.504577 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:43.504540    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"]
Sep 17 14:50:43.525691 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:43.525663    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:50:54.351790 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:54.351757    2565 scope.go:117] "RemoveContainer" containerID="a25dd9d6ea21d5b89442274bb920010f006a9c690f7f269136d9449100786973"
Sep 17 14:50:54.360122 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:54.360088    2565 scope.go:117] "RemoveContainer" containerID="4b2d3d78f3f43d7fbaee4129c9cb0e8fcdb02c0e400609770e290fdad7c5132f"
Sep 17 14:50:54.367415 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:54.367386    2565 scope.go:117] "RemoveContainer" containerID="9399d67af56bb1277e6685ef8cf98c78c0a1c59fbfe7c42dcfdfb37e87081240"
Sep 17 14:50:56.940489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:56.940435    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq" podUID="2a34cad8-fe71-4be5-80ab-e07dcf1568d5" containerName="main" containerID="cri-o://706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e" gracePeriod=45
Sep 17 14:50:57.204168 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.204113    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:50:57.324089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324062    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tmp-dir\") pod \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") "
Sep 17 14:50:57.324212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324098    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-home\") pod \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") "
Sep 17 14:50:57.324212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324139    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tls-certs\") pod \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") "
Sep 17 14:50:57.324212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324175    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2d7tq\" (UniqueName: \"kubernetes.io/projected/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-kube-api-access-2d7tq\") pod \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") "
Sep 17 14:50:57.324212 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324196    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-model-cache\") pod \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") "
Sep 17 14:50:57.324456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324216    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-dshm\") pod \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\" (UID: \"2a34cad8-fe71-4be5-80ab-e07dcf1568d5\") "
Sep 17 14:50:57.324456 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324344    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a34cad8-fe71-4be5-80ab-e07dcf1568d5" (UID: "2a34cad8-fe71-4be5-80ab-e07dcf1568d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:57.324562 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-home" (OuterVolumeSpecName: "home") pod "2a34cad8-fe71-4be5-80ab-e07dcf1568d5" (UID: "2a34cad8-fe71-4be5-80ab-e07dcf1568d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:57.324562 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324490    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a34cad8-fe71-4be5-80ab-e07dcf1568d5" (UID: "2a34cad8-fe71-4be5-80ab-e07dcf1568d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:57.324656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324591    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:57.324656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324610    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:57.324656 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.324623    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:57.326350 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.326327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-dshm" (OuterVolumeSpecName: "dshm") pod "2a34cad8-fe71-4be5-80ab-e07dcf1568d5" (UID: "2a34cad8-fe71-4be5-80ab-e07dcf1568d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:57.326433 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.326358    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a34cad8-fe71-4be5-80ab-e07dcf1568d5" (UID: "2a34cad8-fe71-4be5-80ab-e07dcf1568d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:50:57.326487 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.326457    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-kube-api-access-2d7tq" (OuterVolumeSpecName: "kube-api-access-2d7tq") pod "2a34cad8-fe71-4be5-80ab-e07dcf1568d5" (UID: "2a34cad8-fe71-4be5-80ab-e07dcf1568d5"). InnerVolumeSpecName "kube-api-access-2d7tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:50:57.425767 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.425743    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:57.425767 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.425764    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d7tq\" (UniqueName: \"kubernetes.io/projected/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-kube-api-access-2d7tq\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:57.425893 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.425772    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a34cad8-fe71-4be5-80ab-e07dcf1568d5-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:57.543013 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.542989    2565 generic.go:356] "Generic (PLEG): container finished" podID="2a34cad8-fe71-4be5-80ab-e07dcf1568d5" containerID="706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e" exitCode=0
Sep 17 14:50:57.543111 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.543023    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"
Sep 17 14:50:57.543111 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.543072    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq" event={"ID":"2a34cad8-fe71-4be5-80ab-e07dcf1568d5","Type":"ContainerDied","Data":"706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e"}
Sep 17 14:50:57.543198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.543105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq" event={"ID":"2a34cad8-fe71-4be5-80ab-e07dcf1568d5","Type":"ContainerDied","Data":"a93932130124a96bfd4b0a35527ea45bb9510662f873d119e77a19c111decd7d"}
Sep 17 14:50:57.543198 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.543130    2565 scope.go:117] "RemoveContainer" containerID="706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e"
Sep 17 14:50:57.551089 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.551073    2565 scope.go:117] "RemoveContainer" containerID="706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e"
Sep 17 14:50:57.551369 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:50:57.551351    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e\": container with ID starting with 706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e not found: ID does not exist" containerID="706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e"
Sep 17 14:50:57.551436 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.551375    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e"} err="failed to get container status \"706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e\": rpc error: code = NotFound desc = could not find container \"706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e\": container with ID starting with 706d36e1807e1c8624fd0088b8a30fbe107d82d57dc6e2e449693c18b1041e8e not found: ID does not exist"
Sep 17 14:50:57.560421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.560400    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"]
Sep 17 14:50:57.566912 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:57.566884    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c443cafd-kserve-7bcc854b47-dvkbq"]
Sep 17 14:50:58.523798 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.523752    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8" podUID="db84faad-83b8-4db8-b1e5-4e4c12b79357" containerName="main" containerID="cri-o://f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0" gracePeriod=45
Sep 17 14:50:58.771780 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.771756    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:50:58.836965 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.836940    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-tmp-dir\") pod \"db84faad-83b8-4db8-b1e5-4e4c12b79357\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") "
Sep 17 14:50:58.837097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.836996    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfjwv\" (UniqueName: \"kubernetes.io/projected/db84faad-83b8-4db8-b1e5-4e4c12b79357-kube-api-access-jfjwv\") pod \"db84faad-83b8-4db8-b1e5-4e4c12b79357\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") "
Sep 17 14:50:58.837097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837047    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db84faad-83b8-4db8-b1e5-4e4c12b79357-tls-certs\") pod \"db84faad-83b8-4db8-b1e5-4e4c12b79357\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") "
Sep 17 14:50:58.837097 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837086    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-model-cache\") pod \"db84faad-83b8-4db8-b1e5-4e4c12b79357\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") "
Sep 17 14:50:58.837222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837156    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-dshm\") pod \"db84faad-83b8-4db8-b1e5-4e4c12b79357\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") "
Sep 17 14:50:58.837222 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837186    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-home\") pod \"db84faad-83b8-4db8-b1e5-4e4c12b79357\" (UID: \"db84faad-83b8-4db8-b1e5-4e4c12b79357\") "
Sep 17 14:50:58.837357 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837228    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "db84faad-83b8-4db8-b1e5-4e4c12b79357" (UID: "db84faad-83b8-4db8-b1e5-4e4c12b79357"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:58.837489 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837442    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-model-cache" (OuterVolumeSpecName: "model-cache") pod "db84faad-83b8-4db8-b1e5-4e4c12b79357" (UID: "db84faad-83b8-4db8-b1e5-4e4c12b79357"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:58.837594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837516    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-home" (OuterVolumeSpecName: "home") pod "db84faad-83b8-4db8-b1e5-4e4c12b79357" (UID: "db84faad-83b8-4db8-b1e5-4e4c12b79357"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:58.837594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837531    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-tmp-dir\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:58.837594 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.837551    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-model-cache\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:58.839221 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.839198    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db84faad-83b8-4db8-b1e5-4e4c12b79357-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "db84faad-83b8-4db8-b1e5-4e4c12b79357" (UID: "db84faad-83b8-4db8-b1e5-4e4c12b79357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 14:50:58.839334 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.839241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-dshm" (OuterVolumeSpecName: "dshm") pod "db84faad-83b8-4db8-b1e5-4e4c12b79357" (UID: "db84faad-83b8-4db8-b1e5-4e4c12b79357"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 14:50:58.839334 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.839273    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db84faad-83b8-4db8-b1e5-4e4c12b79357-kube-api-access-jfjwv" (OuterVolumeSpecName: "kube-api-access-jfjwv") pod "db84faad-83b8-4db8-b1e5-4e4c12b79357" (UID: "db84faad-83b8-4db8-b1e5-4e4c12b79357"). InnerVolumeSpecName "kube-api-access-jfjwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 14:50:58.938599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.938579    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfjwv\" (UniqueName: \"kubernetes.io/projected/db84faad-83b8-4db8-b1e5-4e4c12b79357-kube-api-access-jfjwv\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:58.938599 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.938598    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/db84faad-83b8-4db8-b1e5-4e4c12b79357-tls-certs\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:58.938728 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.938607    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-dshm\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:58.938728 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:58.938616    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/db84faad-83b8-4db8-b1e5-4e4c12b79357-home\") on node \"ip-10-0-128-175.ec2.internal\" DevicePath \"\""
Sep 17 14:50:59.520102 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.520077    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a34cad8-fe71-4be5-80ab-e07dcf1568d5" path="/var/lib/kubelet/pods/2a34cad8-fe71-4be5-80ab-e07dcf1568d5/volumes"
Sep 17 14:50:59.561514 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.561483    2565 generic.go:356] "Generic (PLEG): container finished" podID="db84faad-83b8-4db8-b1e5-4e4c12b79357" containerID="f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0" exitCode=0
Sep 17 14:50:59.561897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.561540    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"
Sep 17 14:50:59.561897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.561566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8" event={"ID":"db84faad-83b8-4db8-b1e5-4e4c12b79357","Type":"ContainerDied","Data":"f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0"}
Sep 17 14:50:59.561897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.561618    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8" event={"ID":"db84faad-83b8-4db8-b1e5-4e4c12b79357","Type":"ContainerDied","Data":"d44f25c5073a514bb4d1b2da90e6e7e188489ead495b977f57792221ef76262d"}
Sep 17 14:50:59.561897 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.561643    2565 scope.go:117] "RemoveContainer" containerID="f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0"
Sep 17 14:50:59.570012 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.569995    2565 scope.go:117] "RemoveContainer" containerID="f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0"
Sep 17 14:50:59.570285 ip-10-0-128-175 kubenswrapper[2565]: E0917 14:50:59.570248    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0\": container with ID starting with f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0 not found: ID does not exist" containerID="f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0"
Sep 17 14:50:59.570353 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.570287    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0"} err="failed to get container status \"f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0\": rpc error: code = NotFound desc = could not find container \"f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0\": container with ID starting with f56f037d68d31e71920d7e34925773e3cfa1368d3ac200da5b66e6c3b41e29a0 not found: ID does not exist"
Sep 17 14:50:59.577501 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.577471    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"]
Sep 17 14:50:59.581388 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:50:59.581368    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c443cafd-kserve-8665496987-t25s8"]
Sep 17 14:51:01.519665 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:01.519631    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db84faad-83b8-4db8-b1e5-4e4c12b79357" path="/var/lib/kubelet/pods/db84faad-83b8-4db8-b1e5-4e4c12b79357/volumes"
Sep 17 14:51:41.782646 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:41.782603    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kndh5_34172f79-3f60-4ccf-919a-deb390f30b02/manager/0.log"
Sep 17 14:51:42.491987 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:42.491961    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-569db487f9-x9mlw_59172329-6d9b-48e6-85f1-6ba9226f6cee/copy-manifests/0.log"
Sep 17 14:51:42.498151 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:42.498125    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-569db487f9-x9mlw_59172329-6d9b-48e6-85f1-6ba9226f6cee/manager/0.log"
Sep 17 14:51:43.033875 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:43.033849    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827590-85dxf_d315bd8b-df1e-44be-8742-5c0ee0ba9261/cleanup/0.log"
Sep 17 14:51:43.143855 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:43.143823    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827605-88jrq_acf8b174-26bb-48cc-a491-76b415bf09e7/cleanup/0.log"
Sep 17 14:51:44.328210 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:44.328182    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-wr52n_9c6e6391-e1b9-4ae5-ad32-406cda3a137f/kuadrant-console-plugin/0.log"
Sep 17 14:51:44.440841 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:44.440798    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ks5pv_5a5cfc57-b949-45e9-b7c2-478b724b1aed/manager/0.log"
Sep 17 14:51:45.016066 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:45.016025    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2_9de956cc-9510-458d-84db-b1883388a194/istio-proxy/0.log"
Sep 17 14:51:45.469169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:45.469134    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4x7rq_93debedb-47b6-4fd8-a144-c4c629341e42/istio-proxy/1.log"
Sep 17 14:51:45.812173 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:45.812085    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d847c5fc8-txhzg_183850d9-742f-4556-966d-99d1df6cc9a1/router/0.log"
Sep 17 14:51:46.148318 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:46.148214    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-clvpx_7ebba19f-bfe0-4f42-8c3d-fe2de60ce513/main/0.log"
Sep 17 14:51:46.256645 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:46.256612    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-kpms6_b1fe51fb-e6fb-4048-93e6-88d1d98173af/main/0.log"
Sep 17 14:51:46.698271 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:46.698231    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-zxbv8_e101f8ab-94d0-41d6-b81a-39cb52381ddc/main/0.log"
Sep 17 14:51:52.571877 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:52.571828    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:51:52.721409 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:52.721340    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:51:53.731275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:53.731229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:51:53.740432 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:53.740406    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:51:53.744077 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:53.744056    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:51:53.749455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:53.749438    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8f6tm_a6762df1-b224-4466-ad96-74ee1af45c5f/ovn-acl-logging/0.log"
Sep 17 14:51:55.368268 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:55.368231    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-52tc9_c3a2822e-00c7-4ac7-85cc-816306696a1f/global-pull-secret-syncer/0.log"
Sep 17 14:51:55.495975 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:55.495946    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log"
Sep 17 14:51:55.604280 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:51:55.604241    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-175.ec2.internal_4fd9853fb4ec76c374d0d4c456a7e8b2/haproxy/0.log"
Sep 17 14:52:00.580391 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:00.580357    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-wr52n_9c6e6391-e1b9-4ae5-ad32-406cda3a137f/kuadrant-console-plugin/0.log"
Sep 17 14:52:00.621887 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:00.621857    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-ks5pv_5a5cfc57-b949-45e9-b7c2-478b724b1aed/manager/0.log"
Sep 17 14:52:02.466308 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:02.466271    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-nsmsq_bd6a4dfe-df50-416f-9718-022d6c974565/kube-state-metrics/0.log"
Sep 17 14:52:02.483786 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:02.483755    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-nsmsq_bd6a4dfe-df50-416f-9718-022d6c974565/kube-rbac-proxy-main/0.log"
Sep 17 14:52:02.504522 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:02.504500    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-nsmsq_bd6a4dfe-df50-416f-9718-022d6c974565/kube-rbac-proxy-self/0.log"
Sep 17 14:52:02.797863 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:02.797831    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xp9xh_e63b607b-6798-4045-be4a-1084d98b2b08/node-exporter/0.log"
Sep 17 14:52:02.824171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:02.824148    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xp9xh_e63b607b-6798-4045-be4a-1084d98b2b08/kube-rbac-proxy/0.log"
Sep 17 14:52:02.848686 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:02.848658    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-xp9xh_e63b607b-6798-4045-be4a-1084d98b2b08/init-textfile/0.log"
Sep 17 14:52:02.964085 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:02.963999    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1f677c8-e587-4279-a153-bf9180c8fece/prometheus/0.log"
Sep 17 14:52:02.985387 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:02.985364    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1f677c8-e587-4279-a153-bf9180c8fece/config-reloader/0.log"
Sep 17 14:52:03.012835 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.012815    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1f677c8-e587-4279-a153-bf9180c8fece/thanos-sidecar/0.log"
Sep 17 14:52:03.041228 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.041194    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1f677c8-e587-4279-a153-bf9180c8fece/kube-rbac-proxy-web/0.log"
Sep 17 14:52:03.064559 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.064536    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1f677c8-e587-4279-a153-bf9180c8fece/kube-rbac-proxy/0.log"
Sep 17 14:52:03.088169 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.088140    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1f677c8-e587-4279-a153-bf9180c8fece/kube-rbac-proxy-thanos/0.log"
Sep 17 14:52:03.114075 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.114055    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f1f677c8-e587-4279-a153-bf9180c8fece/init-config-reloader/0.log"
Sep 17 14:52:03.228414 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.228338    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-z2d6q_63d3b0a0-ef56-4182-9395-ed221f69ee85/prometheus-operator-admission-webhook/0.log"
Sep 17 14:52:03.384683 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.384649    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c49449dc-gqzzd_aa638175-c8b8-4ae6-8769-31d4ba1b0395/thanos-query/0.log"
Sep 17 14:52:03.406125 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.406097    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c49449dc-gqzzd_aa638175-c8b8-4ae6-8769-31d4ba1b0395/kube-rbac-proxy-web/0.log"
Sep 17 14:52:03.431275 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.431225    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c49449dc-gqzzd_aa638175-c8b8-4ae6-8769-31d4ba1b0395/kube-rbac-proxy/0.log"
Sep 17 14:52:03.452817 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.452788    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c49449dc-gqzzd_aa638175-c8b8-4ae6-8769-31d4ba1b0395/prom-label-proxy/0.log"
Sep 17 14:52:03.478981 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.478906    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c49449dc-gqzzd_aa638175-c8b8-4ae6-8769-31d4ba1b0395/kube-rbac-proxy-rules/0.log"
Sep 17 14:52:03.499216 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:03.499187    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-85c49449dc-gqzzd_aa638175-c8b8-4ae6-8769-31d4ba1b0395/kube-rbac-proxy-metrics/0.log"
Sep 17 14:52:04.842420 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:04.842391    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-xzvqn_f896ec3f-2d7c-46d4-978d-c87c9187a60b/networking-console-plugin/0.log"
Sep 17 14:52:05.469678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.469649    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/1.log"
Sep 17 14:52:05.474323 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.474292    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-dvbhf_73c3fc4e-356f-49f8-a20d-18abf2d5c52a/console-operator/2.log"
Sep 17 14:52:05.488000 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.487972    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"]
Sep 17 14:52:05.488368 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.488355    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db84faad-83b8-4db8-b1e5-4e4c12b79357" containerName="main"
Sep 17 14:52:05.488416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.488370    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="db84faad-83b8-4db8-b1e5-4e4c12b79357" containerName="main"
Sep 17 14:52:05.488416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.488395    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a34cad8-fe71-4be5-80ab-e07dcf1568d5" containerName="main"
Sep 17 14:52:05.488416 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.488400    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a34cad8-fe71-4be5-80ab-e07dcf1568d5" containerName="main"
Sep 17 14:52:05.488518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.488459    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a34cad8-fe71-4be5-80ab-e07dcf1568d5" containerName="main"
Sep 17 14:52:05.488518 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.488467    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="db84faad-83b8-4db8-b1e5-4e4c12b79357" containerName="main"
Sep 17 14:52:05.491580 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.491559    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.494054 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.494029    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xrpqc\"/\"default-dockercfg-vgnzx\""
Sep 17 14:52:05.494171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.494108    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xrpqc\"/\"kube-root-ca.crt\""
Sep 17 14:52:05.494171 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.494121    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xrpqc\"/\"openshift-service-ca.crt\""
Sep 17 14:52:05.500361 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.500336    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"]
Sep 17 14:52:05.614534 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.614496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-podres\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.614723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.614552    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-sys\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.614723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.614611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-lib-modules\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.614723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.614634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghv4s\" (UniqueName: \"kubernetes.io/projected/5f8124f9-062b-4be7-b509-9528ebe2866c-kube-api-access-ghv4s\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.614723 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.614650    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-proc\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-podres\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715661 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-sys\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-lib-modules\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715750    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghv4s\" (UniqueName: \"kubernetes.io/projected/5f8124f9-062b-4be7-b509-9528ebe2866c-kube-api-access-ghv4s\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715771    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-proc\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715812    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-podres\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715831    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-sys\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715868    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-proc\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.715886 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.715880    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5f8124f9-062b-4be7-b509-9528ebe2866c-lib-modules\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.723946 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.723883    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghv4s\" (UniqueName: \"kubernetes.io/projected/5f8124f9-062b-4be7-b509-9528ebe2866c-kube-api-access-ghv4s\") pod \"perf-node-gather-daemonset-mktnf\" (UID: \"5f8124f9-062b-4be7-b509-9528ebe2866c\") " pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:05.801729 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:05.801689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:06.073068 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:06.073040    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"]
Sep 17 14:52:06.074612 ip-10-0-128-175 kubenswrapper[2565]: W0917 14:52:06.074587    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5f8124f9_062b_4be7_b509_9528ebe2866c.slice/crio-9ab8fd6980c8243499360ffac23dd03a70884124062e52f4b98002c2486975eb WatchSource:0}: Error finding container 9ab8fd6980c8243499360ffac23dd03a70884124062e52f4b98002c2486975eb: Status 404 returned error can't find the container with id 9ab8fd6980c8243499360ffac23dd03a70884124062e52f4b98002c2486975eb
Sep 17 14:52:06.101152 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:06.101125    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-588df94cbc-45qn6_d79e67f7-6318-46da-8fe0-fc9a5cf5948a/console/0.log"
Sep 17 14:52:06.902385 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:06.902352    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf" event={"ID":"5f8124f9-062b-4be7-b509-9528ebe2866c","Type":"ContainerStarted","Data":"16a2c4f5b80269a81289e81a7b70b51658bf575b602f57ed811b98ae60a79634"}
Sep 17 14:52:06.902607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:06.902407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf" event={"ID":"5f8124f9-062b-4be7-b509-9528ebe2866c","Type":"ContainerStarted","Data":"9ab8fd6980c8243499360ffac23dd03a70884124062e52f4b98002c2486975eb"}
Sep 17 14:52:06.902607 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:06.902526    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:06.912716 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:06.912676    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-49687_7ed7979b-5dea-4ff5-bc94-1ebb231696d7/volume-data-source-validator/0.log"
Sep 17 14:52:06.921233 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:06.921183    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf" podStartSLOduration=1.921166388 podStartE2EDuration="1.921166388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 14:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 14:52:06.92020491 +0000 UTC m=+2713.853710098" watchObservedRunningTime="2026-09-17 14:52:06.921166388 +0000 UTC m=+2713.854671580"
Sep 17 14:52:08.045367 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:08.045327    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-flvcb_3b860267-277a-4b1b-85b8-850e86d2cb3d/dns/0.log"
Sep 17 14:52:08.069736 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:08.069706    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-flvcb_3b860267-277a-4b1b-85b8-850e86d2cb3d/kube-rbac-proxy/0.log"
Sep 17 14:52:08.198303 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:08.198239    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dxm97_4d20f337-ad17-4d2c-a2d3-e4e7c15b162d/dns-node-resolver/0.log"
Sep 17 14:52:08.747396 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:08.747361    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-86df9b5656-bq6mt_51870341-f4ff-4804-98c3-31864864cc92/registry/0.log"
Sep 17 14:52:08.803455 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:08.803421    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hztvs_f2312773-bed0-40d9-a40f-1b8f822d4c98/node-ca/0.log"
Sep 17 14:52:09.796064 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:09.796032    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2_9de956cc-9510-458d-84db-b1883388a194/istio-proxy/0.log"
Sep 17 14:52:10.662043 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:10.662016    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4x7rq_93debedb-47b6-4fd8-a144-c4c629341e42/istio-proxy/0.log"
Sep 17 14:52:10.664794 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:10.664777    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4x7rq_93debedb-47b6-4fd8-a144-c4c629341e42/istio-proxy/1.log"
Sep 17 14:52:10.792632 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:10.792610    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d847c5fc8-txhzg_183850d9-742f-4556-966d-99d1df6cc9a1/router/0.log"
Sep 17 14:52:11.746471 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:11.746441    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7wvpb_3d84b0d1-637b-439f-a466-1d55f375e01f/serve-healthcheck-canary/0.log"
Sep 17 14:52:12.387182 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:12.387151    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-fbr4k_50c4e844-c224-42f8-9d28-6f5c733038e8/insights-operator/0.log"
Sep 17 14:52:12.387666 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:12.387646    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-fbr4k_50c4e844-c224-42f8-9d28-6f5c733038e8/insights-operator/1.log"
Sep 17 14:52:12.568916 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:12.568890    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtkwv_85f6b33e-7505-4478-907f-f75fa8d8b07a/kube-rbac-proxy/0.log"
Sep 17 14:52:12.589296 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:12.589272    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtkwv_85f6b33e-7505-4478-907f-f75fa8d8b07a/exporter/0.log"
Sep 17 14:52:12.609907 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:12.609882    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mtkwv_85f6b33e-7505-4478-907f-f75fa8d8b07a/extractor/0.log"
Sep 17 14:52:12.917422 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:12.917394    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xrpqc/perf-node-gather-daemonset-mktnf"
Sep 17 14:52:15.241657 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:15.241593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-kndh5_34172f79-3f60-4ccf-919a-deb390f30b02/manager/0.log"
Sep 17 14:52:15.710801 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:15.710776    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-569db487f9-x9mlw_59172329-6d9b-48e6-85f1-6ba9226f6cee/manager/0.log"
Sep 17 14:52:15.729944 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:15.729920    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-569db487f9-x9mlw_59172329-6d9b-48e6-85f1-6ba9226f6cee/copy-manifests/0.log"
Sep 17 14:52:16.099678 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:16.099650    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2_9de956cc-9510-458d-84db-b1883388a194/istio-proxy/0.log"
Sep 17 14:52:16.955280 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:16.955184    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77fdd77d78-n87hl_4631013b-cd4a-46c2-b1d1-e53523819969/manager/0.log"
Sep 17 14:52:16.957071 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:16.957051    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4x7rq_93debedb-47b6-4fd8-a144-c4c629341e42/istio-proxy/0.log"
Sep 17 14:52:16.959901 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:16.959882    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4x7rq_93debedb-47b6-4fd8-a144-c4c629341e42/istio-proxy/1.log"
Sep 17 14:52:16.981034 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:16.981011    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-d2f6h_1b2a2bee-4d16-4a8a-b2d0-16b0b498eb4f/openshift-lws-operator/0.log"
Sep 17 14:52:17.037574 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:17.037557    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d847c5fc8-txhzg_183850d9-742f-4556-966d-99d1df6cc9a1/router/0.log"
Sep 17 14:52:18.487589 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:18.487559    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-hl7q2_9de956cc-9510-458d-84db-b1883388a194/istio-proxy/0.log"
Sep 17 14:52:19.357069 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:19.357027    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4x7rq_93debedb-47b6-4fd8-a144-c4c629341e42/istio-proxy/0.log"
Sep 17 14:52:19.360763 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:19.360740    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-4x7rq_93debedb-47b6-4fd8-a144-c4c629341e42/istio-proxy/1.log"
Sep 17 14:52:19.453421 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:19.453380    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7d847c5fc8-txhzg_183850d9-742f-4556-966d-99d1df6cc9a1/router/0.log"
Sep 17 14:52:22.325662 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:22.325626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-46lvw_8c3a0f5c-65b9-46f6-a43d-08b7b7427fef/konnectivity-agent/0.log" limit=1024
Sep 17 14:52:22.385078 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:22.385046    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-chdpt_ebf0ae63-0557-403a-a3ee-13969694249d/kube-storage-version-migrator-operator/1.log"
Sep 17 14:52:22.385719 ip-10-0-128-175 kubenswrapper[2565]: I0917 14:52:22.385697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-chdpt_ebf0ae63-0557-403a-a3ee-13969694249d/kube-storage-version-migrator-operator/0.log"