Sep 07 21:01:46.786031 ip-10-0-128-28 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 21:01:47.195989 ip-10-0-128-28 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 21:01:47.195989 ip-10-0-128-28 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 21:01:47.195989 ip-10-0-128-28 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 21:01:47.195989 ip-10-0-128-28 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 21:01:47.195989 ip-10-0-128-28 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 21:01:47.197042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.196958    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 21:01:47.200759 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200743    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 21:01:47.200759 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200758    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200762    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200766    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200769    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200772    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200777    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200781    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200784    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200787    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200791    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200794    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200797    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200805    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200808    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200811    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200814    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200816    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200819    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200822    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 21:01:47.200847 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200825    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200827    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200830    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200832    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200835    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200837    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200840    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200843    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200845    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200848    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200851    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200854    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200857    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200860    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200862    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200865    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200868    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200870    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200873    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200875    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 21:01:47.201470 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200878    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200880    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200883    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200890    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200893    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200895    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200898    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200900    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200903    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200906    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200908    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200911    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200913    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200916    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200919    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200922    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200925    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200927    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200930    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 21:01:47.202178 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200933    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200936    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200939    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200941    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200944    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200949    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200953    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200956    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200959    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200962    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200964    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200967    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200971    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200975    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200977    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200981    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200983    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200986    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200988    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 21:01:47.202720 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200991    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200994    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200996    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.200999    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201002    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201005    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201008    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201010    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201014    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201016    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201019    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201021    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201024    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201026    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201029    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201032    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201034    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201037    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201040    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201042    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.201045    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 21:01:47.203186 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203166    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203178    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203182    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203186    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203189    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203192    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203196    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203199    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203203    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203206    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203209    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203211    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203214    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203217    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203220    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203222    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203225    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203228    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203230    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203233    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 21:01:47.203718 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203236    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203238    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203241    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203243    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203245    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203248    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203251    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203253    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203267    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203271    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203273    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203277    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203283    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203286    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203289    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203292    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203295    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203297    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203300    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 21:01:47.204264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203303    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203307    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203309    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203314    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203317    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203320    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203322    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203325    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203327    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203330    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203333    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203335    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203338    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203341    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203343    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203346    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203349    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203351    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203353    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203356    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 21:01:47.204747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203358    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203361    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203363    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203366    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203368    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203372    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203375    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203378    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203381    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203383    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203386    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203388    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203391    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203393    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203396    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203399    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203401    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203404    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203406    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203409    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 21:01:47.205241 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203412    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203415    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203417    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203420    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203423    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203426    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203428    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203431    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203433    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203436    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203438    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203441    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203444    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203446    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203449    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203453    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203456    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203460    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203463    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 21:01:47.205747 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203467    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203834    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203840    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203843    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203846    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203849    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203852    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203855    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203858    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203860    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203863    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203866    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203869    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203871    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203874    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203876    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203879    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203882    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203884    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203887    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 21:01:47.206212 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203890    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203892    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203894    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203897    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203899    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203902    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203904    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203907    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203909    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203913    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203917    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203919    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203922    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203925    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203927    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203930    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203932    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203935    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203937    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203940    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 21:01:47.206707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203942    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203945    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203947    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203950    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203953    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203956    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203959    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203962    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203964    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203969    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203972    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203975    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203978    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203981    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203983    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203986    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203990    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203993    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203996    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 21:01:47.207195 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.203999    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204002    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204004    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204008    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204010    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204013    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204015    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204018    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204021    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204024    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204026    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204029    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204031    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204034    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204036    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204039    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204041    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204044    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204048    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 21:01:47.207650 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204052    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204055    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204057    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204060    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204062    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204065    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204068    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204070    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204073    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204075    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204078    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204081    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204083    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204086    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204088    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204090    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204094    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204097    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204099    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204102    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 21:01:47.208221 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204104    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204107    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204184    2565 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204222    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204228    2565 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204233    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204237    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204240    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204246    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204250    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204253    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204256    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204259    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204262    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204267    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204270    2565 flags.go:64] FLAG: --cgroup-root=""
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204273    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204276    2565 flags.go:64] FLAG: --client-ca-file=""
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204280    2565 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204283    2565 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204287    2565 flags.go:64] FLAG: --cluster-domain=""
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204290    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204294    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204297    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 21:01:47.208726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204300    2565 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204308    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204311    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204314    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204319    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204322    2565 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204325    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204328    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204331    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204334    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204338    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204341    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204344    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204347    2565 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204350    2565 flags.go:64] FLAG: --enable-server="true"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204353    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204358    2565 flags.go:64] FLAG: --event-burst="100"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204361    2565 flags.go:64] FLAG: --event-qps="50"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204364    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204367    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204370    2565 flags.go:64] FLAG: --eviction-hard=""
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204373    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204376    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204379    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204382    2565 flags.go:64] FLAG: --eviction-soft=""
Sep 07 21:01:47.209320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204385    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204388    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204392    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204395    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204398    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204400    2565 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204403    2565 flags.go:64] FLAG: --feature-gates=""
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204407    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204410    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204413    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204416    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204420    2565 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204423    2565 flags.go:64] FLAG: --help="false"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204426    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204429    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204432    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204435    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204438    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204442    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204445    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204448    2565 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204450    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204453    2565 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 21:01:47.209968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204458    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204461    2565 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204464    2565 flags.go:64] FLAG: --kube-reserved=""
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204466    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204469    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204472    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204475    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204478    2565 flags.go:64] FLAG: --lock-file=""
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204480    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204483    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204486    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204491    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204495    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204498    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204500    2565 flags.go:64] FLAG: --logging-format="text"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204503    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204506    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204509    2565 flags.go:64] FLAG: --manifest-url=""
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204512    2565 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204516    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204520    2565 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204525    2565 flags.go:64] FLAG: --max-pods="110"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204527    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204531    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204534    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 21:01:47.210556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204537    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204539    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204542    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204545    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204553    2565 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204555    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204558    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204563    2565 flags.go:64] FLAG: --pod-cidr=""
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204566    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204573    2565 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204576    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204579    2565 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204582    2565 flags.go:64] FLAG: --port="10250"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204585    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204588    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b3008962a8ec5e67"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204591    2565 flags.go:64] FLAG: --qos-reserved=""
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204593    2565 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204596    2565 flags.go:64] FLAG: --register-node="true"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204599    2565 flags.go:64] FLAG: --register-with-taints=""
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204603    2565 flags.go:64] FLAG: --registry-burst="10"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204606    2565 flags.go:64] FLAG: --registry-qps="5"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204609    2565 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204611    2565 flags.go:64] FLAG: --reserved-memory=""
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204615    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204618    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 21:01:47.211232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204621    2565 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204624    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204626    2565 flags.go:64] FLAG: --runonce="false"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204630    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204634    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204636    2565 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204639    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204642    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204645    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204649    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204665    2565 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204668    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204671    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204673    2565 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204678    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204681    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204684    2565 flags.go:64] FLAG: --system-cgroups=""
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204687    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204692    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204695    2565 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204698    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204703    2565 flags.go:64] FLAG: --tls-min-version=""
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204705    2565 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204708    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204711    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 21:01:47.211855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204714    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204717    2565 flags.go:64] FLAG: --v="2"
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204720    2565 flags.go:64] FLAG: --version="false"
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204727    2565 flags.go:64] FLAG: --vmodule=""
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204731    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.204734    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204883    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204887    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204891    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204894    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204899    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204902    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204905    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204907    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204910    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204913    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204916    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204918    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204921    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204923    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204926    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 21:01:47.212500 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204930    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204933    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204936    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204939    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204941    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204944    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204947    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204950    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204952    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204954    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204957    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204959    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204962    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204965    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204967    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204970    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204973    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204975    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204978    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204980    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 21:01:47.213041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204983    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204987    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204989    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204992    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204995    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.204997    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205000    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205002    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205005    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205008    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205010    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205013    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205029    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205031    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205034    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205036    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205039    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205044    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205047    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 21:01:47.213583 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205050    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205054    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205058    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205061    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205063    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205066    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205069    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205071    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205074    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205077    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205080    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205082    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205085    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205087    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205091    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205094    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205096    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205099    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205102    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205105    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 21:01:47.214069 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205107    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205110    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205112    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205115    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205125    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205129    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205132    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205134    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205137    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205141    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205144    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205147    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205150    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205153    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205156    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205158    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205161    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205164    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205166    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 21:01:47.214557 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205169    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205171    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205174    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205177    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205180    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.205183    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.205842    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.212642    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.212671    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.213626    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.213634    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213704    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213710    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213714    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213717    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 21:01:47.215042 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213720    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213723    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213726    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213729    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213732    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213735    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213738    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213741    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213743    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213746    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213749    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213752    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213754    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213757    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213760    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213763    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213765    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213768    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213770    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 21:01:47.215423 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213773    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213775    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213778    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213781    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213783    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213786    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213790    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213793    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213796    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213798    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213801    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213804    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213806    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213809    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213812    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213814    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213817    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213820    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213822    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213825    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 21:01:47.215900 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213828    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213830    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213833    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213836    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213838    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213841    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213844    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213846    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213849    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213852    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213855    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213858    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213861    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213863    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213866    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213869    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213871    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213874    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213877    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213881    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 21:01:47.216393 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213883    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213886    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213889    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213892    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213894    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213897    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213899    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213903    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213907    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213909    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213912    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213914    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213917    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213920    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213922    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213925    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213928    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213930    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213933    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 21:01:47.216951 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213937    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213941    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213944    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213947    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213950    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213953    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213956    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213959    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213963    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213966    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213969    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213973    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213976    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213979    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213982    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213985    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 21:01:47.217430 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.213988    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.213993    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214118    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214124    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214127    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214130    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214133    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214136    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214139    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214142    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214144    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214147    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214150    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214152    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 21:01:47.217826 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214155    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214157    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214160    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214162    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214165    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214167    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214170    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214173    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214175    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214178    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214182    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214185    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214188    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214191    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214195    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214198    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214201    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214204    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214206    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 21:01:47.218173 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214209    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214211    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214215    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214217    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214220    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214222    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214225    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214228    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214230    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214233    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214235    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214238    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214240    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214243    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214245    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214248    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214250    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214253    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214255    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214258    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 21:01:47.218625 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214261    2565 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214263    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214266    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214268    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214271    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214273    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214276    2565 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214279    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214281    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214283    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214292    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214295    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214298    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214300    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214303    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214305    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214308    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214310    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214312    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214315    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 21:01:47.219199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214317    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214320    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214322    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214325    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214328    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214330    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214333    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214336    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214339    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214341    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214344    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214346    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214349    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214352    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214354    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214357    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214359    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214362    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214365    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214367    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 21:01:47.219706 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214370    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214374    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214377    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214387    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214390    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214393    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214396    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:47.214399    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.214404    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.214513    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.216324    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.217280    2565 server.go:1019] "Starting client certificate rotation"
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.217392    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 21:01:47.220205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.217465    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 21:01:47.220520 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.219950    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 21:01:47.221305 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.221289    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 21:01:47.231938 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.231921    2565 log.go:25] "Validated CRI v1 runtime API"
Sep 07 21:01:47.235718 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.235700    2565 log.go:25] "Validated CRI v1 image API"
Sep 07 21:01:47.236541 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.236521    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 21:01:47.239890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.239869    2565 fs.go:135] Filesystem UUIDs: map[23bb5b1d-1ada-4420-a854-2e519164b855:/dev/nvme0n1p4 2f0d1475-0ba9-41b8-99ef-ca2357a4e0c8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 07 21:01:47.239982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.239889    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 21:01:47.244336 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.244316    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 21:01:47.245507 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.245398    2565 manager.go:216] Machine: {Timestamp:2026-09-07 21:01:47.243400851 +0000 UTC m=+0.341236163 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098540 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28742642e42c81c1a305a3a860af59 SystemUUID:ec287426-42e4-2c81-c1a3-05a3a860af59 BootID:75bc681b-7242-4c57-9e46-ee24e83d2f04 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e2:cf:1f:fb:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e2:cf:1f:fb:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:63:0c:2e:ab:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 21:01:47.245507 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.245505    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 21:01:47.245680 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.245646    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 21:01:47.249383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.249361    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 21:01:47.249512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.249386    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 21:01:47.249970 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.249960    2565 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 21:01:47.250007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.249972    2565 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 21:01:47.250007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.249980    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 21:01:47.250486 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.250477    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 21:01:47.251228 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.251218    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 21:01:47.254684 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.254673    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 21:01:47.254800 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.254791    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 21:01:47.254835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.254824    2565 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 21:01:47.254835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.254833    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 21:01:47.254897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.254846    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 21:01:47.254897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.254855    2565 kubelet.go:398] "Adding apiserver pod source"
Sep 07 21:01:47.254897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.254864    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 21:01:47.255395 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.255377    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 21:01:47.255445 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.255405    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 21:01:47.257701 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.257685    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 21:01:47.260292 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.260279    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 21:01:47.260349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.260311    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 21:01:47.261328 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261315    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 21:01:47.261384 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261332    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 21:01:47.261384 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261338    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 21:01:47.261384 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261344    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 21:01:47.261384 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261352    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 21:01:47.261384 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261360    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 21:01:47.261527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261424    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 21:01:47.261527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261430    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 21:01:47.261527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261437    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 21:01:47.261527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261442    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 21:01:47.261527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261456    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 21:01:47.261527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.261463    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 21:01:47.263404 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.263393    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 21:01:47.263404 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.263403    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 21:01:47.266084 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.266064    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r87rr"
Sep 07 21:01:47.266667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.266627    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 21:01:47.266752 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.266700    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 21:01:47.266804 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.266762    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 21:01:47.267348 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.267335    2565 server.go:1268] "Started kubelet"
Sep 07 21:01:47.267456 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.267419    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 21:01:47.267510 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.267471    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 21:01:47.267555 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.267539    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 21:01:47.268130 ip-10-0-128-28 systemd[1]: Started Kubernetes Kubelet.
Sep 07 21:01:47.268627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.268564    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 21:01:47.273713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.273690    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r87rr"
Sep 07 21:01:47.274112 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.274095    2565 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 21:01:47.275011 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.274984    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 21:01:47.276795 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.275732    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-28.ec2.internal.18d325ae3c9763bd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-28.ec2.internal,UID:ip-10-0-128-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-28.ec2.internal,},FirstTimestamp:2026-09-07 21:01:47.267310525 +0000 UTC m=+0.365145839,LastTimestamp:2026-09-07 21:01:47.267310525 +0000 UTC m=+0.365145839,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-28.ec2.internal,}"
Sep 07 21:01:47.278113 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.278096    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 21:01:47.278715 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.278699    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 21:01:47.278802 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.278780    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:47.279252 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.279234    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 21:01:47.279252 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.279233    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 21:01:47.279377 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.279270    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 21:01:47.279377 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.279329    2565 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 21:01:47.279377 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.279337    2565 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 21:01:47.280481 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.280467    2565 factory.go:153] Registering CRI-O factory
Sep 07 21:01:47.280560 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.280508    2565 factory.go:223] Registration of the crio container factory successfully
Sep 07 21:01:47.280624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.280578    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 21:01:47.280624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.280589    2565 factory.go:55] Registering systemd factory
Sep 07 21:01:47.280624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.280597    2565 factory.go:223] Registration of the systemd container factory successfully
Sep 07 21:01:47.280624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.280613    2565 factory.go:103] Registering Raw factory
Sep 07 21:01:47.280624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.280626    2565 manager.go:1196] Started watching for new ooms in manager
Sep 07 21:01:47.281033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.281010    2565 manager.go:349] Starting recovery of all containers
Sep 07 21:01:47.288646 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.288628    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 21:01:47.291046 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.291027    2565 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-28.ec2.internal\" not found" node="ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.293332 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.293316    2565 manager.go:354] Recovery completed
Sep 07 21:01:47.297597 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.297584    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 21:01:47.299524 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.299511    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 21:01:47.299590 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.299534    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 21:01:47.299590 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.299545    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientPID"
Sep 07 21:01:47.300090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.300075    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 21:01:47.300090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.300086    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 21:01:47.300243 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.300088    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 21:01:47.300243 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.300160    2565 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 21:01:47.300839 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.300822    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 21:01:47.300890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.300851    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 21:01:47.300890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.300866    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientPID"
Sep 07 21:01:47.302233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.302210    2565 policy_none.go:49] "None policy: Start"
Sep 07 21:01:47.302233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.302227    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 21:01:47.302233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.302237    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 21:01:47.304111 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.304098    2565 policy_none.go:47] "Start"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.335274    2565 manager.go:337] "Starting Device Plugin manager"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.335419    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.335429    2565 server.go:87] "Starting device plugin registration server"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.335647    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.335673    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.336040    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.336146    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.336155    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.338615    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.338718    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.339042    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.340938    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.340977    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.340998    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.341059    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 07 21:01:47.353504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.343088    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 21:01:47.435819 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.435787    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 21:01:47.436714 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.436694    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 21:01:47.436808 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.436721    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 21:01:47.436808 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.436732    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientPID"
Sep 07 21:01:47.436808 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.436751    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.441819 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.441802    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal"]
Sep 07 21:01:47.441878 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.441860    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 21:01:47.442540 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.442526    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 21:01:47.442608 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.442549    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 21:01:47.442608 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.442558    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientPID"
Sep 07 21:01:47.442999 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.442985    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.443054 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.443008    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-28.ec2.internal\": node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:47.444837 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.444825    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 21:01:47.444996 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.444983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.445054 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.445016    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 21:01:47.445464 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.445449    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 21:01:47.445540 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.445473    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 21:01:47.445540 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.445483    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientPID"
Sep 07 21:01:47.445540 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.445510    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 21:01:47.445540 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.445535    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 21:01:47.445761 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.445548    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientPID"
Sep 07 21:01:47.447903 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.447834    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.447903 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.447862    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 21:01:47.448485 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.448468    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 21:01:47.448566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.448494    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 21:01:47.448566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.448503    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeHasSufficientPID"
Sep 07 21:01:47.456674 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.456647    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:47.463227 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.463213    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-28.ec2.internal\" not found" node="ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.466957 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.466940    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-28.ec2.internal\" not found" node="ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.480714 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.480697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2d8773881ab781b00886710f6952dd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal\" (UID: \"d2d8773881ab781b00886710f6952dd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.480783 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.480723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2d8773881ab781b00886710f6952dd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal\" (UID: \"d2d8773881ab781b00886710f6952dd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.480783 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.480739    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a426b2798cb1a9b52d39c4792ff821c-config\") pod \"kube-apiserver-proxy-ip-10-0-128-28.ec2.internal\" (UID: \"1a426b2798cb1a9b52d39c4792ff821c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.557183 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.557157    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:47.581196 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.581181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2d8773881ab781b00886710f6952dd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal\" (UID: \"d2d8773881ab781b00886710f6952dd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.581277 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.581204    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2d8773881ab781b00886710f6952dd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal\" (UID: \"d2d8773881ab781b00886710f6952dd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.581277 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.581223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a426b2798cb1a9b52d39c4792ff821c-config\") pod \"kube-apiserver-proxy-ip-10-0-128-28.ec2.internal\" (UID: \"1a426b2798cb1a9b52d39c4792ff821c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.581371 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.581285    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a426b2798cb1a9b52d39c4792ff821c-config\") pod \"kube-apiserver-proxy-ip-10-0-128-28.ec2.internal\" (UID: \"1a426b2798cb1a9b52d39c4792ff821c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.581371 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.581289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d2d8773881ab781b00886710f6952dd2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal\" (UID: \"d2d8773881ab781b00886710f6952dd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.581371 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.581299    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2d8773881ab781b00886710f6952dd2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal\" (UID: \"d2d8773881ab781b00886710f6952dd2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.657998 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.657969    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:47.758751 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.758680    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:47.764838 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.764821    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.770740 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:47.770720    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:47.859533 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.859505    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:47.960001 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:47.959969    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:48.060527 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:48.060465    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:49.177379 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:49.177353    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:49.233876 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.233851    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 21:01:49.234460 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.233997    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 21:01:49.234460 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.234030    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 21:01:49.278255 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:49.278226    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:49.294602 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.294573    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 20:56:48 +0000 UTC" deadline="2028-02-07 21:39:55.697826465 +0000 UTC"
Sep 07 21:01:49.294682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.294604    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12432h38m6.403226737s"
Sep 07 21:01:49.358536 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:49.358503    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a426b2798cb1a9b52d39c4792ff821c.slice/crio-7c7b3f170d09ab4fef65168ef489fe80f4e373859805b36776af1615fce391a3 WatchSource:0}: Error finding container 7c7b3f170d09ab4fef65168ef489fe80f4e373859805b36776af1615fce391a3: Status 404 returned error can't find the container with id 7c7b3f170d09ab4fef65168ef489fe80f4e373859805b36776af1615fce391a3
Sep 07 21:01:49.359199 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:49.359177    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d8773881ab781b00886710f6952dd2.slice/crio-877ce0a18fd005d5e18d1457d6f13a69d54eac760053496203f6391ba09d2bcd WatchSource:0}: Error finding container 877ce0a18fd005d5e18d1457d6f13a69d54eac760053496203f6391ba09d2bcd: Status 404 returned error can't find the container with id 877ce0a18fd005d5e18d1457d6f13a69d54eac760053496203f6391ba09d2bcd
Sep 07 21:01:49.364362 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.364335    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 21:01:49.379038 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:49.379018    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:49.479170 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:49.479148    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:49.479781 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.479765    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 21:01:49.580061 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:49.580033    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:49.680529 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:49.680473    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-28.ec2.internal\" not found"
Sep 07 21:01:49.770958 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.770930    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 21:01:49.795924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.795898    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:49.806086 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.806066    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 21:01:49.807061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.807048    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal"
Sep 07 21:01:49.816674 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:49.816658    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 21:01:50.272795 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.272763    2565 apiserver.go:52] "Watching apiserver"
Sep 07 21:01:50.279642 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.279609    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 21:01:50.282796 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.282768    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-9tlvn","kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal","openshift-image-registry/node-ca-h94gv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal","openshift-multus/multus-additional-cni-plugins-6hds9","openshift-network-diagnostics/network-check-target-ssk4q","openshift-network-operator/iptables-alerter-svtjh","kube-system/konnectivity-agent-w9r8n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f","openshift-cluster-node-tuning-operator/tuned-c85bs","openshift-dns/node-resolver-22b4b","openshift-multus/multus-dpq7j","openshift-multus/network-metrics-daemon-xbp8h","openshift-ovn-kubernetes/ovnkube-node-gxfll"]
Sep 07 21:01:50.285193 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.285168    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.287304 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.287283    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:50.287402 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.287376    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:01:50.287586 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.287561    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 21:01:50.287640 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.287614    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 21:01:50.287817 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.287804    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-28ww5\""
Sep 07 21:01:50.290429 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.290407    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.293500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.292685    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 21:01:50.293500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.293001    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 21:01:50.293500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.293043    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bshp8\""
Sep 07 21:01:50.293500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.293098    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 21:01:50.294651 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.294629    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.294738 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.294697    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 21:01:50.297130 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.296820    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 21:01:50.297130 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.296852    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.297130 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.296923    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 21:01:50.297681 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.297660    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 21:01:50.297761 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.297693    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4fjgz\""
Sep 07 21:01:50.297761 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.297708    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 21:01:50.297876 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.297859    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 21:01:50.298019 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.298000    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 21:01:50.298907 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.298889    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 21:01:50.299278 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.299259    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:50.299397 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.299326    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:01:50.299545 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.299513    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 21:01:50.299625 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.299595    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 21:01:50.299730 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.299705    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 21:01:50.299730 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.299724    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 21:01:50.300022 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.300002    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pls2p\""
Sep 07 21:01:50.302104 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.301728    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.303573 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-var-lib-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.303691 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303621    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-ovn\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.303762 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303713    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l9hlf\""
Sep 07 21:01:50.303762 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303733    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-cni-netd\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.303864 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.303997 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-ovnkube-config\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.303997 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303955    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-kubernetes\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.303997 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303993    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysctl-conf\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.304233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304005    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 21:01:50.304233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303888    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 21:01:50.304233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304048    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-systemd-units\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.303875    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 21:01:50.304233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304151    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-run-netns\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-node-log\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-log-socket\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-cni-bin\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-ovnkube-script-lib\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-host\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97nng\" (UniqueName: \"kubernetes.io/projected/02eff29a-6dd4-495c-84b1-97c952ad2b0b-kube-api-access-97nng\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304348    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304376    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysconfig\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02eff29a-6dd4-495c-84b1-97c952ad2b0b-host\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304478    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-slash\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304535    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/008d9424-b077-4d6f-afae-2682be76cd70-ovn-node-metrics-cert\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-system-cni-dir\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysctl-d\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-var-lib-kubelet\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.304669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304640    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-env-overrides\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75d6h\" (UniqueName: \"kubernetes.io/projected/04e701e8-8302-4722-865f-c23db8b82045-kube-api-access-75d6h\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304695    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-lib-modules\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304738    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-modprobe-d\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304772    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-kubelet\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304821    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-etc-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304858    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-run\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02eff29a-6dd4-495c-84b1-97c952ad2b0b-serviceca\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304912    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-cnibin\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304944    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-sys\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.304983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-tuned\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.305008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh664\" (UniqueName: \"kubernetes.io/projected/c2eaac87-a88b-4c27-a225-05028ce893d1-kube-api-access-mh664\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.305033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-systemd\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.305056    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l4ht\" (UniqueName: \"kubernetes.io/projected/008d9424-b077-4d6f-afae-2682be76cd70-kube-api-access-5l4ht\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.305085    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.305567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.305107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-systemd\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.306317 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.305134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2eaac87-a88b-4c27-a225-05028ce893d1-tmp\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.306317 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.305169    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmpx9\" (UniqueName: \"kubernetes.io/projected/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-kube-api-access-jmpx9\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:50.306904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.306658    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 21:01:50.306904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.306692    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 21:01:50.306904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.306725    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 21:01:50.306904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.306747    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pbgdv\""
Sep 07 21:01:50.307139 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.306930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:01:50.307417 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.307393    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 21:01:50.309064 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.309036    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.309178 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.309089    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:01:50.309887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.309634    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 21:01:50.310062 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.310000    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 21:01:50.310201 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.310184    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w7lbz\""
Sep 07 21:01:50.311627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.311609    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.313857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.313640    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 21:01:50.313857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.313790    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-n6824\""
Sep 07 21:01:50.313857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.313817    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 21:01:50.314698 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.314680    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.316948 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.316924    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4whtr\""
Sep 07 21:01:50.317077 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.316976    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 21:01:50.327134 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.327117    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5kfhx"
Sep 07 21:01:50.335824 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.335803    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5kfhx"
Sep 07 21:01:50.362793 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.362717    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal" event={"ID":"d2d8773881ab781b00886710f6952dd2","Type":"ContainerStarted","Data":"877ce0a18fd005d5e18d1457d6f13a69d54eac760053496203f6391ba09d2bcd"}
Sep 07 21:01:50.364189 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.364164    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal" event={"ID":"1a426b2798cb1a9b52d39c4792ff821c","Type":"ContainerStarted","Data":"7c7b3f170d09ab4fef65168ef489fe80f4e373859805b36776af1615fce391a3"}
Sep 07 21:01:50.397480 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.397459    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 21:01:50.405511 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405485    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-ovn\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-cni-netd\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-ovn\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-ovnkube-config\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-cni-netd\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-system-cni-dir\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.405645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405627    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-systemd-units\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405636    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-system-cni-dir\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405678    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-systemd-units\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405689    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-run-netns\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405714    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-cni-bin\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-ovnkube-script-lib\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-socket-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405790    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-run-netns\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405823    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-cni-bin\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-hostroot\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405936    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cef8007-e9b4-42de-a19f-5133be487f62-iptables-alerter-script\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.405988 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.405984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-host\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406071    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-run-ovn-kubernetes\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-device-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406097    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-host\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406114    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jgms\" (UniqueName: \"kubernetes.io/projected/4cef8007-e9b4-42de-a19f-5133be487f62-kube-api-access-2jgms\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c541d1bb-269e-436b-95da-76cde9b0c2e8-konnectivity-ca\") pod \"konnectivity-agent-w9r8n\" (UID: \"c541d1bb-269e-436b-95da-76cde9b0c2e8\") " pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406153    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn7mp\" (UniqueName: \"kubernetes.io/projected/57134fda-ea48-44a8-920e-b435bb868eb4-kube-api-access-qn7mp\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-slash\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12c4533d-f234-4b23-b343-c3cebf3279e9-dbus\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-cni-bin\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406232    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c541d1bb-269e-436b-95da-76cde9b0c2e8-agent-certs\") pod \"konnectivity-agent-w9r8n\" (UID: \"c541d1bb-269e-436b-95da-76cde9b0c2e8\") " pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406262    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysctl-d\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-ovnkube-config\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406292    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-env-overrides\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406293    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-ovnkube-script-lib\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406332    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.406667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-lib-modules\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-modprobe-d\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysctl-d\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406397    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-cni-binary-copy\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57134fda-ea48-44a8-920e-b435bb868eb4-hosts-file\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406445    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-run\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406455    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-lib-modules\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02eff29a-6dd4-495c-84b1-97c952ad2b0b-serviceca\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406474    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-tuned\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406493    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-run\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh664\" (UniqueName: \"kubernetes.io/projected/c2eaac87-a88b-4c27-a225-05028ce893d1-kube-api-access-mh664\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-modprobe-d\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8j2\" (UniqueName: \"kubernetes.io/projected/14f12cfe-3682-461a-83d0-9590e2ba971e-kube-api-access-jj8j2\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.406518    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-slash\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/883e3628-8e19-4c1a-933c-d7209b86b4cd-cni-binary-copy\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.407495 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.406611    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs podName:2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8 nodeName:}" failed. No retries permitted until 2026-09-07 21:01:50.906591147 +0000 UTC m=+2.988118487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs") pod "network-metrics-daemon-xbp8h" (UID: "2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406626    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/008d9424-b077-4d6f-afae-2682be76cd70-env-overrides\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406632    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmgw7\" (UniqueName: \"kubernetes.io/projected/883e3628-8e19-4c1a-933c-d7209b86b4cd-kube-api-access-dmgw7\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2eaac87-a88b-4c27-a225-05028ce893d1-tmp\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmpx9\" (UniqueName: \"kubernetes.io/projected/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-kube-api-access-jmpx9\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-socket-dir-parent\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406857    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-cni-multus\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/02eff29a-6dd4-495c-84b1-97c952ad2b0b-serviceca\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-multus-certs\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-kubernetes\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysctl-conf\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406975    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-node-log\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406991    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407034    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-log-socket\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.406998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-log-socket\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407037    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-node-log\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.408303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407031    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-kubernetes\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407072    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-registration-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407155    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysctl-conf\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12c4533d-f234-4b23-b343-c3cebf3279e9-kubelet-config\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407222    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97nng\" (UniqueName: \"kubernetes.io/projected/02eff29a-6dd4-495c-84b1-97c952ad2b0b-kube-api-access-97nng\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-system-cni-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-cni-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407292    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-k8s-cni-cncf-io\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407316    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-netns\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407338    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cef8007-e9b4-42de-a19f-5133be487f62-host-slash\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407360    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57134fda-ea48-44a8-920e-b435bb868eb4-tmp-dir\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysconfig\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02eff29a-6dd4-495c-84b1-97c952ad2b0b-host\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407456    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/008d9424-b077-4d6f-afae-2682be76cd70-ovn-node-metrics-cert\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.408912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-conf-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-var-lib-kubelet\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75d6h\" (UniqueName: \"kubernetes.io/projected/04e701e8-8302-4722-865f-c23db8b82045-kube-api-access-75d6h\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-cnibin\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-kubelet\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407601    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-daemon-config\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-kubelet\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407630    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-sysconfig\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407633    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407651    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-etc-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407675    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407578    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04e701e8-8302-4722-865f-c23db8b82045-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407725    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-etc-selinux\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407775    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-cnibin\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-etc-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-host-kubelet\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-etc-kubernetes\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.409566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407841    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-sys\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407913    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.407954    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/02eff29a-6dd4-495c-84b1-97c952ad2b0b-host\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-systemd\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408036    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l4ht\" (UniqueName: \"kubernetes.io/projected/008d9424-b077-4d6f-afae-2682be76cd70-kube-api-access-5l4ht\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408035    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-var-lib-kubelet\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408089    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-run-systemd\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408113    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-sys-fs\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-systemd\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408239    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-tuning-conf-dir\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408265    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-var-lib-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408289    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-systemd\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408305    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2eaac87-a88b-4c27-a225-05028ce893d1-sys\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04e701e8-8302-4722-865f-c23db8b82045-cnibin\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.410208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.408395    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/008d9424-b077-4d6f-afae-2682be76cd70-var-lib-openvswitch\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.410670 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.410358    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2eaac87-a88b-4c27-a225-05028ce893d1-etc-tuned\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.410670 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.410419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2eaac87-a88b-4c27-a225-05028ce893d1-tmp\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.410670 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.410495    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/008d9424-b077-4d6f-afae-2682be76cd70-ovn-node-metrics-cert\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.416337 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.415997    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l4ht\" (UniqueName: \"kubernetes.io/projected/008d9424-b077-4d6f-afae-2682be76cd70-kube-api-access-5l4ht\") pod \"ovnkube-node-gxfll\" (UID: \"008d9424-b077-4d6f-afae-2682be76cd70\") " pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.416337 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.416065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh664\" (UniqueName: \"kubernetes.io/projected/c2eaac87-a88b-4c27-a225-05028ce893d1-kube-api-access-mh664\") pod \"tuned-c85bs\" (UID: \"c2eaac87-a88b-4c27-a225-05028ce893d1\") " pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.416337 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.416291    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75d6h\" (UniqueName: \"kubernetes.io/projected/04e701e8-8302-4722-865f-c23db8b82045-kube-api-access-75d6h\") pod \"multus-additional-cni-plugins-6hds9\" (UID: \"04e701e8-8302-4722-865f-c23db8b82045\") " pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.416928 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.416908    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97nng\" (UniqueName: \"kubernetes.io/projected/02eff29a-6dd4-495c-84b1-97c952ad2b0b-kube-api-access-97nng\") pod \"node-ca-h94gv\" (UID: \"02eff29a-6dd4-495c-84b1-97c952ad2b0b\") " pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.417222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.417198    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmpx9\" (UniqueName: \"kubernetes.io/projected/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-kube-api-access-jmpx9\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:50.509260 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509217    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-registration-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.509260 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12c4533d-f234-4b23-b343-c3cebf3279e9-kubelet-config\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509279    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-system-cni-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-cni-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-k8s-cni-cncf-io\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-netns\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509360    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-system-cni-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509376    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cef8007-e9b4-42de-a19f-5133be487f62-host-slash\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509348    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-registration-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509346    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/12c4533d-f234-4b23-b343-c3cebf3279e9-kubelet-config\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509387    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-k8s-cni-cncf-io\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57134fda-ea48-44a8-920e-b435bb868eb4-tmp-dir\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-netns\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509390    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-cni-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509452    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-conf-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509475    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-cnibin\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.509496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509497    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-kubelet\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-conf-dir\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509439    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4cef8007-e9b4-42de-a19f-5133be487f62-host-slash\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-daemon-config\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509573    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-cnibin\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509610    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-etc-selinux\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-kubelet\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-etc-kubernetes\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509662    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509701    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-etc-kubernetes\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509708    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-etc-selinux\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509664    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-sys-fs\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509743    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57134fda-ea48-44a8-920e-b435bb868eb4-tmp-dir\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-socket-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509754    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-sys-fs\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-hostroot\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cef8007-e9b4-42de-a19f-5133be487f62-iptables-alerter-script\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.510044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-device-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgms\" (UniqueName: \"kubernetes.io/projected/4cef8007-e9b4-42de-a19f-5133be487f62-kube-api-access-2jgms\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509889    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-hostroot\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509906    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-device-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509929    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c541d1bb-269e-436b-95da-76cde9b0c2e8-konnectivity-ca\") pod \"konnectivity-agent-w9r8n\" (UID: \"c541d1bb-269e-436b-95da-76cde9b0c2e8\") " pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.509974    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn7mp\" (UniqueName: \"kubernetes.io/projected/57134fda-ea48-44a8-920e-b435bb868eb4-kube-api-access-qn7mp\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510003    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12c4533d-f234-4b23-b343-c3cebf3279e9-dbus\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-daemon-config\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-cni-bin\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510049    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c541d1bb-269e-436b-95da-76cde9b0c2e8-agent-certs\") pod \"konnectivity-agent-w9r8n\" (UID: \"c541d1bb-269e-436b-95da-76cde9b0c2e8\") " pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510119    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-cni-bin\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.510156    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57134fda-ea48-44a8-920e-b435bb868eb4-hosts-file\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510200    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8j2\" (UniqueName: \"kubernetes.io/projected/14f12cfe-3682-461a-83d0-9590e2ba971e-kube-api-access-jj8j2\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.510219    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret podName:12c4533d-f234-4b23-b343-c3cebf3279e9 nodeName:}" failed. No retries permitted until 2026-09-07 21:01:51.010201135 +0000 UTC m=+3.091728468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret") pod "global-pull-secret-syncer-9tlvn" (UID: "12c4533d-f234-4b23-b343-c3cebf3279e9") : object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510244    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/12c4533d-f234-4b23-b343-c3cebf3279e9-dbus\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:50.510956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510269    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57134fda-ea48-44a8-920e-b435bb868eb4-hosts-file\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510298    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/883e3628-8e19-4c1a-933c-d7209b86b4cd-cni-binary-copy\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510323    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmgw7\" (UniqueName: \"kubernetes.io/projected/883e3628-8e19-4c1a-933c-d7209b86b4cd-kube-api-access-dmgw7\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510385    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4cef8007-e9b4-42de-a19f-5133be487f62-iptables-alerter-script\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510393    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c541d1bb-269e-436b-95da-76cde9b0c2e8-konnectivity-ca\") pod \"konnectivity-agent-w9r8n\" (UID: \"c541d1bb-269e-436b-95da-76cde9b0c2e8\") " pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-socket-dir-parent\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510578    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-cni-multus\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-multus-certs\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510618    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-multus-socket-dir-parent\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510675    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-run-multus-certs\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510676    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/883e3628-8e19-4c1a-933c-d7209b86b4cd-host-var-lib-cni-multus\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510758    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/14f12cfe-3682-461a-83d0-9590e2ba971e-socket-dir\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.511548 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.510871    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/883e3628-8e19-4c1a-933c-d7209b86b4cd-cni-binary-copy\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.512756 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.512733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c541d1bb-269e-436b-95da-76cde9b0c2e8-agent-certs\") pod \"konnectivity-agent-w9r8n\" (UID: \"c541d1bb-269e-436b-95da-76cde9b0c2e8\") " pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:01:50.516526 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.516504    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 21:01:50.516526 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.516526    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 21:01:50.516667 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.516538    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9ccs4 for pod openshift-network-diagnostics/network-check-target-ssk4q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:50.516667 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.516656    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4 podName:188765a8-5bc2-48c7-b6ca-c63aa8e4adbf nodeName:}" failed. No retries permitted until 2026-09-07 21:01:51.016638422 +0000 UTC m=+3.098165743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9ccs4" (UniqueName: "kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4") pod "network-check-target-ssk4q" (UID: "188765a8-5bc2-48c7-b6ca-c63aa8e4adbf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:50.518874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.518818    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jgms\" (UniqueName: \"kubernetes.io/projected/4cef8007-e9b4-42de-a19f-5133be487f62-kube-api-access-2jgms\") pod \"iptables-alerter-svtjh\" (UID: \"4cef8007-e9b4-42de-a19f-5133be487f62\") " pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.518874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.518829    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmgw7\" (UniqueName: \"kubernetes.io/projected/883e3628-8e19-4c1a-933c-d7209b86b4cd-kube-api-access-dmgw7\") pod \"multus-dpq7j\" (UID: \"883e3628-8e19-4c1a-933c-d7209b86b4cd\") " pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.519029 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.518958    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn7mp\" (UniqueName: \"kubernetes.io/projected/57134fda-ea48-44a8-920e-b435bb868eb4-kube-api-access-qn7mp\") pod \"node-resolver-22b4b\" (UID: \"57134fda-ea48-44a8-920e-b435bb868eb4\") " pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.519270 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.519251    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8j2\" (UniqueName: \"kubernetes.io/projected/14f12cfe-3682-461a-83d0-9590e2ba971e-kube-api-access-jj8j2\") pod \"aws-ebs-csi-driver-node-kc64f\" (UID: \"14f12cfe-3682-461a-83d0-9590e2ba971e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.598266 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.598235    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c85bs"
Sep 07 21:01:50.605567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.605543    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h94gv"
Sep 07 21:01:50.616466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.616444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:01:50.623200 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.623183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-6hds9"
Sep 07 21:01:50.628942 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.628922    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f"
Sep 07 21:01:50.635863 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.635591    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-svtjh"
Sep 07 21:01:50.645203 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.645186    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:01:50.651831 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.651813    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-22b4b"
Sep 07 21:01:50.657398 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.657383    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dpq7j"
Sep 07 21:01:50.703399 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.703376    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 21:01:50.814379 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.814356    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 21:01:50.914044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:50.913954    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:50.914191 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.914107    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:50.914191 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:50.914167    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs podName:2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8 nodeName:}" failed. No retries permitted until 2026-09-07 21:01:51.914152636 +0000 UTC m=+3.995679956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs") pod "network-metrics-daemon-xbp8h" (UID: "2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:51.014320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.014284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:51.014476 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:51.014459    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:51.014530 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:51.014519    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret podName:12c4533d-f234-4b23-b343-c3cebf3279e9 nodeName:}" failed. No retries permitted until 2026-09-07 21:01:52.014504473 +0000 UTC m=+4.096031791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret") pod "global-pull-secret-syncer-9tlvn" (UID: "12c4533d-f234-4b23-b343-c3cebf3279e9") : object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:51.115040 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.115006    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:51.115199 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:51.115167    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 21:01:51.115199 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:51.115186    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 21:01:51.115199 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:51.115195    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9ccs4 for pod openshift-network-diagnostics/network-check-target-ssk4q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:51.115352 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:51.115245    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4 podName:188765a8-5bc2-48c7-b6ca-c63aa8e4adbf nodeName:}" failed. No retries permitted until 2026-09-07 21:01:52.115229819 +0000 UTC m=+4.196757158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ccs4" (UniqueName: "kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4") pod "network-check-target-ssk4q" (UID: "188765a8-5bc2-48c7-b6ca-c63aa8e4adbf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:51.315151 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.315125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14f12cfe_3682_461a_83d0_9590e2ba971e.slice/crio-d412a6d80ff53be07e76b1d1f1f60cf29a218ff4f603ec84ca787cc6ec1e8e52 WatchSource:0}: Error finding container d412a6d80ff53be07e76b1d1f1f60cf29a218ff4f603ec84ca787cc6ec1e8e52: Status 404 returned error can't find the container with id d412a6d80ff53be07e76b1d1f1f60cf29a218ff4f603ec84ca787cc6ec1e8e52
Sep 07 21:01:51.316274 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.316190    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod883e3628_8e19_4c1a_933c_d7209b86b4cd.slice/crio-ef63a4604d4b9e2c977174da66aebe4cb14b0509586be1c8543e192ae6e5dc4f WatchSource:0}: Error finding container ef63a4604d4b9e2c977174da66aebe4cb14b0509586be1c8543e192ae6e5dc4f: Status 404 returned error can't find the container with id ef63a4604d4b9e2c977174da66aebe4cb14b0509586be1c8543e192ae6e5dc4f
Sep 07 21:01:51.318063 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.317926    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc541d1bb_269e_436b_95da_76cde9b0c2e8.slice/crio-35050990bda367c9951a6f68f5102e762222c9438d8b124e9dccc457238a1bfc WatchSource:0}: Error finding container 35050990bda367c9951a6f68f5102e762222c9438d8b124e9dccc457238a1bfc: Status 404 returned error can't find the container with id 35050990bda367c9951a6f68f5102e762222c9438d8b124e9dccc457238a1bfc
Sep 07 21:01:51.320367 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.320345    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eff29a_6dd4_495c_84b1_97c952ad2b0b.slice/crio-2187287f3db9832aa3321bb972880dfde7276d224cf002d4893024b709d70f46 WatchSource:0}: Error finding container 2187287f3db9832aa3321bb972880dfde7276d224cf002d4893024b709d70f46: Status 404 returned error can't find the container with id 2187287f3db9832aa3321bb972880dfde7276d224cf002d4893024b709d70f46
Sep 07 21:01:51.321048 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.321024    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57134fda_ea48_44a8_920e_b435bb868eb4.slice/crio-1fbbe557b1960dcabeb67766d6dce8666fff81fef82d7be892bcefb01368f975 WatchSource:0}: Error finding container 1fbbe557b1960dcabeb67766d6dce8666fff81fef82d7be892bcefb01368f975: Status 404 returned error can't find the container with id 1fbbe557b1960dcabeb67766d6dce8666fff81fef82d7be892bcefb01368f975
Sep 07 21:01:51.321744 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.321361    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2eaac87_a88b_4c27_a225_05028ce893d1.slice/crio-0407cd51a2ef47b2ebe406e1893677fbd34c31a92c3102743596d74be1b3da42 WatchSource:0}: Error finding container 0407cd51a2ef47b2ebe406e1893677fbd34c31a92c3102743596d74be1b3da42: Status 404 returned error can't find the container with id 0407cd51a2ef47b2ebe406e1893677fbd34c31a92c3102743596d74be1b3da42
Sep 07 21:01:51.322950 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.322924    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cef8007_e9b4_42de_a19f_5133be487f62.slice/crio-d2a87f47e29b45698f12451ab75a86ced1c6700921daa8d230a9226d4cfe853a WatchSource:0}: Error finding container d2a87f47e29b45698f12451ab75a86ced1c6700921daa8d230a9226d4cfe853a: Status 404 returned error can't find the container with id d2a87f47e29b45698f12451ab75a86ced1c6700921daa8d230a9226d4cfe853a
Sep 07 21:01:51.324730 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.324672    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e701e8_8302_4722_865f_c23db8b82045.slice/crio-78b6c2efa87e68dec3412c91884f630848f3f8c88240c62860d93272a57b1b34 WatchSource:0}: Error finding container 78b6c2efa87e68dec3412c91884f630848f3f8c88240c62860d93272a57b1b34: Status 404 returned error can't find the container with id 78b6c2efa87e68dec3412c91884f630848f3f8c88240c62860d93272a57b1b34
Sep 07 21:01:51.325915 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:01:51.325893    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008d9424_b077_4d6f_afae_2682be76cd70.slice/crio-8f34bc93315a0a21ffa8a5bad5a1137330f67c2d7d49560fa4796db1f095689c WatchSource:0}: Error finding container 8f34bc93315a0a21ffa8a5bad5a1137330f67c2d7d49560fa4796db1f095689c: Status 404 returned error can't find the container with id 8f34bc93315a0a21ffa8a5bad5a1137330f67c2d7d49560fa4796db1f095689c
Sep 07 21:01:51.338887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.338838    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 20:56:50 +0000 UTC" deadline="2028-02-08 02:25:00.394939307 +0000 UTC"
Sep 07 21:01:51.338985 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.338888    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12437h23m9.056055698s"
Sep 07 21:01:51.370200 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.370059    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"8f34bc93315a0a21ffa8a5bad5a1137330f67c2d7d49560fa4796db1f095689c"}
Sep 07 21:01:51.371482 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.371462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-svtjh" event={"ID":"4cef8007-e9b4-42de-a19f-5133be487f62","Type":"ContainerStarted","Data":"d2a87f47e29b45698f12451ab75a86ced1c6700921daa8d230a9226d4cfe853a"}
Sep 07 21:01:51.372624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.372602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c85bs" event={"ID":"c2eaac87-a88b-4c27-a225-05028ce893d1","Type":"ContainerStarted","Data":"0407cd51a2ef47b2ebe406e1893677fbd34c31a92c3102743596d74be1b3da42"}
Sep 07 21:01:51.373835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.373795    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h94gv" event={"ID":"02eff29a-6dd4-495c-84b1-97c952ad2b0b","Type":"ContainerStarted","Data":"2187287f3db9832aa3321bb972880dfde7276d224cf002d4893024b709d70f46"}
Sep 07 21:01:51.374930 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.374898    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w9r8n" event={"ID":"c541d1bb-269e-436b-95da-76cde9b0c2e8","Type":"ContainerStarted","Data":"35050990bda367c9951a6f68f5102e762222c9438d8b124e9dccc457238a1bfc"}
Sep 07 21:01:51.376686 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.376668    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dpq7j" event={"ID":"883e3628-8e19-4c1a-933c-d7209b86b4cd","Type":"ContainerStarted","Data":"ef63a4604d4b9e2c977174da66aebe4cb14b0509586be1c8543e192ae6e5dc4f"}
Sep 07 21:01:51.377780 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.377750    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-22b4b" event={"ID":"57134fda-ea48-44a8-920e-b435bb868eb4","Type":"ContainerStarted","Data":"1fbbe557b1960dcabeb67766d6dce8666fff81fef82d7be892bcefb01368f975"}
Sep 07 21:01:51.378900 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.378881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hds9" event={"ID":"04e701e8-8302-4722-865f-c23db8b82045","Type":"ContainerStarted","Data":"78b6c2efa87e68dec3412c91884f630848f3f8c88240c62860d93272a57b1b34"}
Sep 07 21:01:51.380013 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.379996    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f" event={"ID":"14f12cfe-3682-461a-83d0-9590e2ba971e","Type":"ContainerStarted","Data":"d412a6d80ff53be07e76b1d1f1f60cf29a218ff4f603ec84ca787cc6ec1e8e52"}
Sep 07 21:01:51.923377 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:51.923342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:51.923571 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:51.923473    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:51.923571 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:51.923565    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs podName:2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8 nodeName:}" failed. No retries permitted until 2026-09-07 21:01:53.923544655 +0000 UTC m=+6.005071980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs") pod "network-metrics-daemon-xbp8h" (UID: "2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:52.024573 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.024530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:52.024750 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.024732    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:52.024843 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.024814    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret podName:12c4533d-f234-4b23-b343-c3cebf3279e9 nodeName:}" failed. No retries permitted until 2026-09-07 21:01:54.024777062 +0000 UTC m=+6.106304403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret") pod "global-pull-secret-syncer-9tlvn" (UID: "12c4533d-f234-4b23-b343-c3cebf3279e9") : object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:52.126200 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.125539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:52.126200 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.125721    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 21:01:52.126200 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.125741    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 21:01:52.126200 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.125753    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9ccs4 for pod openshift-network-diagnostics/network-check-target-ssk4q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:52.126200 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.125807    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4 podName:188765a8-5bc2-48c7-b6ca-c63aa8e4adbf nodeName:}" failed. No retries permitted until 2026-09-07 21:01:54.125790453 +0000 UTC m=+6.207317787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ccs4" (UniqueName: "kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4") pod "network-check-target-ssk4q" (UID: "188765a8-5bc2-48c7-b6ca-c63aa8e4adbf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:52.339657 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.339543    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 20:56:50 +0000 UTC" deadline="2028-03-30 05:13:40.936074104 +0000 UTC"
Sep 07 21:01:52.339657 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.339599    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13664h11m48.596478714s"
Sep 07 21:01:52.359169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.359067    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:52.359293 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.359180    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:01:52.359709 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.359594    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:52.359709 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.359687    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:01:52.359939 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.359878    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:52.360013 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:52.359997    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:01:52.397820 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.397788    2565 generic.go:356] "Generic (PLEG): container finished" podID="d2d8773881ab781b00886710f6952dd2" containerID="c56003f0b32bc0de8a36a07ab22658852b3e618881c77cdb20958e7ffbd106b2" exitCode=0
Sep 07 21:01:52.398182 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.398067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal" event={"ID":"d2d8773881ab781b00886710f6952dd2","Type":"ContainerDied","Data":"c56003f0b32bc0de8a36a07ab22658852b3e618881c77cdb20958e7ffbd106b2"}
Sep 07 21:01:52.429169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.429141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal" event={"ID":"1a426b2798cb1a9b52d39c4792ff821c","Type":"ContainerStarted","Data":"f8b3c869c7aa762d75adec15cee4a11efe18a887565e60b5ce0a8aab987c3883"}
Sep 07 21:01:52.447764 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:52.447661    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-28.ec2.internal" podStartSLOduration=3.4476425490000002 podStartE2EDuration="3.447642549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:01:52.446864318 +0000 UTC m=+4.528391665" watchObservedRunningTime="2026-09-07 21:01:52.447642549 +0000 UTC m=+4.529169891"
Sep 07 21:01:53.463082 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:53.462943    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal" event={"ID":"d2d8773881ab781b00886710f6952dd2","Type":"ContainerStarted","Data":"28834b8df5c6e06ea8af170ceca6440b241981ecd0b1eb606b20ae37def6384f"}
Sep 07 21:01:53.941277 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:53.941242    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:53.941463 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:53.941374    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:53.941463 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:53.941444    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs podName:2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8 nodeName:}" failed. No retries permitted until 2026-09-07 21:01:57.941417631 +0000 UTC m=+10.022944963 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs") pod "network-metrics-daemon-xbp8h" (UID: "2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:54.042485 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:54.042443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:54.042648 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.042584    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:54.042715 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.042650    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret podName:12c4533d-f234-4b23-b343-c3cebf3279e9 nodeName:}" failed. No retries permitted until 2026-09-07 21:01:58.042630835 +0000 UTC m=+10.124158160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret") pod "global-pull-secret-syncer-9tlvn" (UID: "12c4533d-f234-4b23-b343-c3cebf3279e9") : object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:54.143862 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:54.143825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:54.144032 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.144003    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 21:01:54.144032 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.144022    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 21:01:54.144154 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.144036    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9ccs4 for pod openshift-network-diagnostics/network-check-target-ssk4q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:54.144154 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.144096    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4 podName:188765a8-5bc2-48c7-b6ca-c63aa8e4adbf nodeName:}" failed. No retries permitted until 2026-09-07 21:01:58.144078015 +0000 UTC m=+10.225605357 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ccs4" (UniqueName: "kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4") pod "network-check-target-ssk4q" (UID: "188765a8-5bc2-48c7-b6ca-c63aa8e4adbf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:54.359012 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:54.358490    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:54.359012 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.358615    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:01:54.359012 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:54.358998    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:54.359296 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.359089    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:01:54.359296 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:54.359136    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:54.359296 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:54.359217    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:01:56.358626 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:56.358587    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:56.359072 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:56.358587    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:56.359072 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:56.358713    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:01:56.359072 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:56.358787    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:01:56.359072 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:56.358841    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:56.359072 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:56.358908    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:01:57.977203 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:57.976637    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:57.977203 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:57.976806    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:57.977203 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:57.976867    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs podName:2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:05.976848202 +0000 UTC m=+18.058375526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs") pod "network-metrics-daemon-xbp8h" (UID: "2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:01:58.077032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:58.076988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:58.077222 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.077150    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:58.077222 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.077219    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret podName:12c4533d-f234-4b23-b343-c3cebf3279e9 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:06.077198855 +0000 UTC m=+18.158726180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret") pod "global-pull-secret-syncer-9tlvn" (UID: "12c4533d-f234-4b23-b343-c3cebf3279e9") : object "kube-system"/"original-pull-secret" not registered
Sep 07 21:01:58.177767 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:58.177727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:58.177959 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.177944    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 21:01:58.178035 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.177980    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 21:01:58.178035 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.177993    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9ccs4 for pod openshift-network-diagnostics/network-check-target-ssk4q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:58.178104 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.178057    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4 podName:188765a8-5bc2-48c7-b6ca-c63aa8e4adbf nodeName:}" failed. No retries permitted until 2026-09-07 21:02:06.178035358 +0000 UTC m=+18.259562679 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ccs4" (UniqueName: "kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4") pod "network-check-target-ssk4q" (UID: "188765a8-5bc2-48c7-b6ca-c63aa8e4adbf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:01:58.359995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:58.359533    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:01:58.359995 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.359648    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:01:58.359995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:58.359857    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:01:58.359995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:01:58.359983    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:01:58.360327 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.360029    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:01:58.360327 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:01:58.360101    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:00.357799 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:00.357764    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:00.357799 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:00.357796    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:00.358362 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:00.357764    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:00.358362 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:00.357900    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:00.358362 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:00.358005    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:00.358362 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:00.358089    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:02.358201 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:02.358166    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:02.358699 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:02.358166    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:02.358699 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:02.358306    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:02.358699 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:02.358407    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:02.358699 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:02.358442    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:02.358699 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:02.358517    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:04.358461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:04.358425    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:04.358904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:04.358425    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:04.358904 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:04.358549    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:04.358904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:04.358432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:04.358904 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:04.358662    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:04.358904 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:04.358703    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:06.034071 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:06.034020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:06.034493 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.034177    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:02:06.034493 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.034238    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs podName:2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:22.034220791 +0000 UTC m=+34.115748115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs") pod "network-metrics-daemon-xbp8h" (UID: "2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:02:06.134503 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:06.134465    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:06.134669 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.134607    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 21:02:06.134669 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.134663    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret podName:12c4533d-f234-4b23-b343-c3cebf3279e9 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:22.134648802 +0000 UTC m=+34.216176131 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret") pod "global-pull-secret-syncer-9tlvn" (UID: "12c4533d-f234-4b23-b343-c3cebf3279e9") : object "kube-system"/"original-pull-secret" not registered
Sep 07 21:02:06.235144 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:06.235108    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:06.235310 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.235289    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 21:02:06.235371 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.235314    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 21:02:06.235371 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.235324    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9ccs4 for pod openshift-network-diagnostics/network-check-target-ssk4q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:02:06.235442 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.235381    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4 podName:188765a8-5bc2-48c7-b6ca-c63aa8e4adbf nodeName:}" failed. No retries permitted until 2026-09-07 21:02:22.235362069 +0000 UTC m=+34.316889391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ccs4" (UniqueName: "kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4") pod "network-check-target-ssk4q" (UID: "188765a8-5bc2-48c7-b6ca-c63aa8e4adbf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:02:06.358110 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:06.358077    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:06.358286 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:06.358085    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:06.358286 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.358210    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:06.358286 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:06.358220    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:06.358451 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.358290    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:06.358451 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:06.358384    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:08.358228 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:08.358190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:08.358673 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:08.358290    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:08.358673 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:08.358316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:08.358673 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:08.358426    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:08.358673 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:08.358496    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:08.358673 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:08.358592    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:09.496353 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.496026    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"4b72adce3745729c4bf1f9db6eb419eaf7fe37ac077f776deda10b698be5b38f"}
Sep 07 21:02:09.496726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.496402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"e1eb07d4c973a37610478ddd266e19ffe261df84835ed7f5d9645187c5385b2c"}
Sep 07 21:02:09.496726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.496454    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"84dc6559b7f222f0edfec5ff8ece77d24fb32304f13d5c0d756e96958d50e79a"}
Sep 07 21:02:09.496726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.496488    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"110b5ce177cd546d5bc19235b0ebd47cdad4202db4dc5a72ac03a59798b5be9e"}
Sep 07 21:02:09.497354 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.497334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c85bs" event={"ID":"c2eaac87-a88b-4c27-a225-05028ce893d1","Type":"ContainerStarted","Data":"49971f0300c03dba2643da47186dbbfdfb2134ed5b32aab41bf34f91b7ee6fe8"}
Sep 07 21:02:09.498628 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.498607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h94gv" event={"ID":"02eff29a-6dd4-495c-84b1-97c952ad2b0b","Type":"ContainerStarted","Data":"232b8f609afda4ff96d40106bd5461ced3b182c1bdbee6e982d9f7656d941fa9"}
Sep 07 21:02:09.499792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.499774    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-w9r8n" event={"ID":"c541d1bb-269e-436b-95da-76cde9b0c2e8","Type":"ContainerStarted","Data":"32e6965fd78e1acd3813685dfed49d326aeb3359d5e8057fe4c58767286ea9bd"}
Sep 07 21:02:09.503671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.503648    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dpq7j" event={"ID":"883e3628-8e19-4c1a-933c-d7209b86b4cd","Type":"ContainerStarted","Data":"f028e8b25dbef12234e691ad6e23c391c02f2d499cda6f7e81b4f80c8a8297af"}
Sep 07 21:02:09.504809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.504779    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-22b4b" event={"ID":"57134fda-ea48-44a8-920e-b435bb868eb4","Type":"ContainerStarted","Data":"4b906192576e4903add34855553891e0743502af64ee98c76068a0ac725b4bae"}
Sep 07 21:02:09.506133 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.506114    2565 generic.go:356] "Generic (PLEG): container finished" podID="04e701e8-8302-4722-865f-c23db8b82045" containerID="2ba201e85be7b9cad71c747ea6e0ba3b97e1652a7870b53ac0466d8546239849" exitCode=0
Sep 07 21:02:09.506190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.506140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hds9" event={"ID":"04e701e8-8302-4722-865f-c23db8b82045","Type":"ContainerDied","Data":"2ba201e85be7b9cad71c747ea6e0ba3b97e1652a7870b53ac0466d8546239849"}
Sep 07 21:02:09.507571 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.507555    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f" event={"ID":"14f12cfe-3682-461a-83d0-9590e2ba971e","Type":"ContainerStarted","Data":"496399d4533b4fdaf50a1fa8057ab28172464672fae554af836070d9fbb00469"}
Sep 07 21:02:09.526926 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.526886    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-28.ec2.internal" podStartSLOduration=20.526874033 podStartE2EDuration="20.526874033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:01:53.479206211 +0000 UTC m=+5.560733553" watchObservedRunningTime="2026-09-07 21:02:09.526874033 +0000 UTC m=+21.608401374"
Sep 07 21:02:09.540080 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.540042    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-w9r8n" podStartSLOduration=4.037273175 podStartE2EDuration="21.540030274s" totalImagesPullingTime="17.502757099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:09.539756153 +0000 UTC m=+21.621283494" watchObservedRunningTime="2026-09-07 21:02:09.540030274 +0000 UTC m=+21.621557615"
Sep 07 21:02:09.554993 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.554943    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dpq7j" podStartSLOduration=3.988403886 podStartE2EDuration="21.554930647s" totalImagesPullingTime="17.566526761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:09.55451102 +0000 UTC m=+21.636038361" watchObservedRunningTime="2026-09-07 21:02:09.554930647 +0000 UTC m=+21.636457991"
Sep 07 21:02:09.570070 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.570034    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c85bs" podStartSLOduration=4.02746834 podStartE2EDuration="21.570023583s" totalImagesPullingTime="17.542555243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:09.569883449 +0000 UTC m=+21.651410790" watchObservedRunningTime="2026-09-07 21:02:09.570023583 +0000 UTC m=+21.651550924"
Sep 07 21:02:09.583455 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.583423    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h94gv" podStartSLOduration=4.056308445 podStartE2EDuration="21.58341307s" totalImagesPullingTime="17.527104625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:09.583399778 +0000 UTC m=+21.664927132" watchObservedRunningTime="2026-09-07 21:02:09.58341307 +0000 UTC m=+21.664940402"
Sep 07 21:02:09.597307 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:09.597251    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-22b4b" podStartSLOduration=4.099262617 podStartE2EDuration="21.597239892s" totalImagesPullingTime="17.497977275s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:09.597205338 +0000 UTC m=+21.678732678" watchObservedRunningTime="2026-09-07 21:02:09.597239892 +0000 UTC m=+21.678767233"
Sep 07 21:02:10.358441 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.358409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:10.358575 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:10.358537    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:10.358575 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.358409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:10.358684 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.358576    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:10.358684 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:10.358671    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:10.358789 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:10.358765    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:10.448363 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.448336    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 21:02:10.513764 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.513731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"c9781f42a4e8e1ca51c3a6b211013d9a9b56cf9eb92ba2d2edfb53c59c38dfa1"}
Sep 07 21:02:10.514466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.513859    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"c2988f12267bc58d8d9ff7f4f4471ae06549ebd84a07c291f49d5d9f10f97499"}
Sep 07 21:02:10.515454 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.515429    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-svtjh" event={"ID":"4cef8007-e9b4-42de-a19f-5133be487f62","Type":"ContainerStarted","Data":"9ce3be8d1993c1d497cfda19f443d60cde582395d93446e0bf7a16273477903b"}
Sep 07 21:02:10.517585 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.517549    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f" event={"ID":"14f12cfe-3682-461a-83d0-9590e2ba971e","Type":"ContainerStarted","Data":"7e54982d24c0c4837ccbc9391ff6192b5e3b9ba7ecebd6c77e2e7a32acacb7c3"}
Sep 07 21:02:10.534945 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:10.534904    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-svtjh" podStartSLOduration=4.993433596 podStartE2EDuration="22.534891993s" totalImagesPullingTime="17.541458397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:10.534549007 +0000 UTC m=+22.616076412" watchObservedRunningTime="2026-09-07 21:02:10.534891993 +0000 UTC m=+22.616419333"
Sep 07 21:02:11.363188 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:11.363089    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T21:02:10.448356954Z","UUID":"f107087d-4923-4f36-a9b2-79a4cae37641","Handler":null,"Name":"","Endpoint":""}
Sep 07 21:02:11.364895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:11.364873    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 21:02:11.365033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:11.364906    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 21:02:12.358425 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:12.358230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:12.358899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:12.358230    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:12.358899 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:12.358530    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:12.358899 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:12.358568    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:12.358899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:12.358231    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:12.358899 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:12.358645    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:12.526157 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:12.526116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"fd4f76152593c62d47e9b8e60d6fd463b5d07b57aa7c1e1d3e3aa7e4418c9522"}
Sep 07 21:02:12.528411 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:12.528371    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f" event={"ID":"14f12cfe-3682-461a-83d0-9590e2ba971e","Type":"ContainerStarted","Data":"bdf0daf426036b83de8483c48f6c5123bca0a95b51a4558c6ace76f299a2b615"}
Sep 07 21:02:12.545910 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:12.545866    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kc64f" podStartSLOduration=4.602553179 podStartE2EDuration="24.545851994s" totalImagesPullingTime="19.943298815s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:12.545625046 +0000 UTC m=+24.627152388" watchObservedRunningTime="2026-09-07 21:02:12.545851994 +0000 UTC m=+24.627379334"
Sep 07 21:02:14.336411 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.336236    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:02:14.336890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.336870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:02:14.357653 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.357631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:14.357744 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.357653    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:14.357985 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:14.357739    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:14.357985 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:14.357795    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:14.357985 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.357839    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:14.358136 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:14.357983    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:14.527909 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.527879    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:02:14.528366 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.528349    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-w9r8n"
Sep 07 21:02:14.534433 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.534402    2565 generic.go:356] "Generic (PLEG): container finished" podID="04e701e8-8302-4722-865f-c23db8b82045" containerID="530c0ac6c58d5339dc0a18f05f3f2824bda19de7fa68b11a45a5b7835c263a34" exitCode=0
Sep 07 21:02:14.534535 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.534465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hds9" event={"ID":"04e701e8-8302-4722-865f-c23db8b82045","Type":"ContainerDied","Data":"530c0ac6c58d5339dc0a18f05f3f2824bda19de7fa68b11a45a5b7835c263a34"}
Sep 07 21:02:14.538269 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.538249    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" event={"ID":"008d9424-b077-4d6f-afae-2682be76cd70","Type":"ContainerStarted","Data":"b0e580786a3fb02356fe347bda0477e46fd71693c50c9246fa10b004f68889ed"}
Sep 07 21:02:14.538506 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.538490    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:02:14.553387 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.553368    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:02:14.588521 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:14.588449    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll" podStartSLOduration=8.808819183 podStartE2EDuration="26.588436353s" totalImagesPullingTime="17.77961717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:14.587955869 +0000 UTC m=+26.669483211" watchObservedRunningTime="2026-09-07 21:02:14.588436353 +0000 UTC m=+26.669963694"
Sep 07 21:02:15.542784 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:15.542707    2565 generic.go:356] "Generic (PLEG): container finished" podID="04e701e8-8302-4722-865f-c23db8b82045" containerID="13ec8663465c8e9b1412bb3573e7d4a1de5228285824949e116092a0e0ecde64" exitCode=0
Sep 07 21:02:15.543361 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:15.542789    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hds9" event={"ID":"04e701e8-8302-4722-865f-c23db8b82045","Type":"ContainerDied","Data":"13ec8663465c8e9b1412bb3573e7d4a1de5228285824949e116092a0e0ecde64"}
Sep 07 21:02:15.545942 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:15.545849    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:02:15.546676 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:15.546195    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:02:15.561875 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:15.561855    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:02:16.030687 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:16.030655    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9tlvn"]
Sep 07 21:02:16.030876 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:16.030786    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:16.030945 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:16.030913    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:16.031318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:16.031293    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ssk4q"]
Sep 07 21:02:16.031415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:16.031403    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:16.031525 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:16.031500    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:16.033154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:16.033134    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xbp8h"]
Sep 07 21:02:16.033268 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:16.033223    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:16.033372 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:16.033351    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:17.358118 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:17.357895    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:17.358684 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:17.357942    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:17.358684 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:17.358196    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:17.358684 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:17.358289    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:17.358684 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:17.357980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:17.358684 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:17.358396    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:17.550052 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:17.550020    2565 generic.go:356] "Generic (PLEG): container finished" podID="04e701e8-8302-4722-865f-c23db8b82045" containerID="ed9fd0112931af095ff979af82f9fbceea12b402603a1db5a92f0aff2f5e7f30" exitCode=0
Sep 07 21:02:17.550245 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:17.550109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hds9" event={"ID":"04e701e8-8302-4722-865f-c23db8b82045","Type":"ContainerDied","Data":"ed9fd0112931af095ff979af82f9fbceea12b402603a1db5a92f0aff2f5e7f30"}
Sep 07 21:02:19.358347 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:19.358312    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:19.358765 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:19.358437    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:19.358765 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:19.358446    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:19.358765 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:19.358474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:19.358765 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:19.358572    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:19.358765 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:19.358652    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:21.357770 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.357736    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:21.358296 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.357739    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:21.358296 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:21.357835    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ssk4q" podUID="188765a8-5bc2-48c7-b6ca-c63aa8e4adbf"
Sep 07 21:02:21.358296 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.357848    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:21.358296 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:21.357942    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9tlvn" podUID="12c4533d-f234-4b23-b343-c3cebf3279e9"
Sep 07 21:02:21.358296 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:21.358053    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xbp8h" podUID="2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8"
Sep 07 21:02:21.743886 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.743820    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-28.ec2.internal" event="NodeReady"
Sep 07 21:02:21.744027 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.743933    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 21:02:21.788290 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.788262    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z6twk"]
Sep 07 21:02:21.808109 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.808076    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ggj9j"]
Sep 07 21:02:21.808237 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.808220    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:21.810722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.810704    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 21:02:21.810834 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.810720    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 21:02:21.810834 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.810712    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7njqf\""
Sep 07 21:02:21.828193 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.828121    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z6twk"]
Sep 07 21:02:21.828193 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.828182    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggj9j"
Sep 07 21:02:21.828342 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.828229    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggj9j"]
Sep 07 21:02:21.830505 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.830489    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 21:02:21.830699 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.830685    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 21:02:21.830868 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.830849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 21:02:21.830868 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.830864    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jrq2d\""
Sep 07 21:02:21.948320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.948300    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917948a6-64e5-49c9-9bc9-31f75632da0b-tmp-dir\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:21.948413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.948331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54304c6d-a18b-46f3-929b-58f0acfe7751-cert\") pod \"ingress-canary-ggj9j\" (UID: \"54304c6d-a18b-46f3-929b-58f0acfe7751\") " pod="openshift-ingress-canary/ingress-canary-ggj9j"
Sep 07 21:02:21.948413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.948357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ct2\" (UniqueName: \"kubernetes.io/projected/54304c6d-a18b-46f3-929b-58f0acfe7751-kube-api-access-j7ct2\") pod \"ingress-canary-ggj9j\" (UID: \"54304c6d-a18b-46f3-929b-58f0acfe7751\") " pod="openshift-ingress-canary/ingress-canary-ggj9j"
Sep 07 21:02:21.948502 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.948414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/917948a6-64e5-49c9-9bc9-31f75632da0b-config-volume\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:21.948502 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.948473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg9kq\" (UniqueName: \"kubernetes.io/projected/917948a6-64e5-49c9-9bc9-31f75632da0b-kube-api-access-gg9kq\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:21.948567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:21.948513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.049653 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.049629    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:22.049748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.049662    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg9kq\" (UniqueName: \"kubernetes.io/projected/917948a6-64e5-49c9-9bc9-31f75632da0b-kube-api-access-gg9kq\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.049748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.049693    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.049748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.049728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917948a6-64e5-49c9-9bc9-31f75632da0b-tmp-dir\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.049748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.049742    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54304c6d-a18b-46f3-929b-58f0acfe7751-cert\") pod \"ingress-canary-ggj9j\" (UID: \"54304c6d-a18b-46f3-929b-58f0acfe7751\") " pod="openshift-ingress-canary/ingress-canary-ggj9j"
Sep 07 21:02:22.049995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.049758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ct2\" (UniqueName: \"kubernetes.io/projected/54304c6d-a18b-46f3-929b-58f0acfe7751-kube-api-access-j7ct2\") pod \"ingress-canary-ggj9j\" (UID: \"54304c6d-a18b-46f3-929b-58f0acfe7751\") " pod="openshift-ingress-canary/ingress-canary-ggj9j"
Sep 07 21:02:22.049995 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.049765    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:02:22.049995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.049781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/917948a6-64e5-49c9-9bc9-31f75632da0b-config-volume\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.049995 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.049823    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs podName:2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:54.049802643 +0000 UTC m=+66.131329961 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs") pod "network-metrics-daemon-xbp8h" (UID: "2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 21:02:22.049995 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.049862    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 21:02:22.049995 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.049923    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls podName:917948a6-64e5-49c9-9bc9-31f75632da0b nodeName:}" failed. No retries permitted until 2026-09-07 21:02:22.549907112 +0000 UTC m=+34.631434438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls") pod "dns-default-z6twk" (UID: "917948a6-64e5-49c9-9bc9-31f75632da0b") : secret "dns-default-metrics-tls" not found
Sep 07 21:02:22.050260 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.050127    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/917948a6-64e5-49c9-9bc9-31f75632da0b-tmp-dir\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.050361 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.050342    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/917948a6-64e5-49c9-9bc9-31f75632da0b-config-volume\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.053576 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.053558    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/54304c6d-a18b-46f3-929b-58f0acfe7751-cert\") pod \"ingress-canary-ggj9j\" (UID: \"54304c6d-a18b-46f3-929b-58f0acfe7751\") " pod="openshift-ingress-canary/ingress-canary-ggj9j"
Sep 07 21:02:22.059645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.059625    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg9kq\" (UniqueName: \"kubernetes.io/projected/917948a6-64e5-49c9-9bc9-31f75632da0b-kube-api-access-gg9kq\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.059645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.059641    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ct2\" (UniqueName: \"kubernetes.io/projected/54304c6d-a18b-46f3-929b-58f0acfe7751-kube-api-access-j7ct2\") pod \"ingress-canary-ggj9j\" (UID: \"54304c6d-a18b-46f3-929b-58f0acfe7751\") " pod="openshift-ingress-canary/ingress-canary-ggj9j"
Sep 07 21:02:22.136313 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.136266    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ggj9j"
Sep 07 21:02:22.150932 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.150905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:22.151066 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.151039    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 21:02:22.151131 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.151099    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret podName:12c4533d-f234-4b23-b343-c3cebf3279e9 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:54.15108243 +0000 UTC m=+66.232609768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret") pod "global-pull-secret-syncer-9tlvn" (UID: "12c4533d-f234-4b23-b343-c3cebf3279e9") : object "kube-system"/"original-pull-secret" not registered
Sep 07 21:02:22.251656 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.251625    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:22.251797 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.251781    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 21:02:22.251838 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.251802    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 21:02:22.251838 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.251811    2565 projected.go:196] Error preparing data for projected volume kube-api-access-9ccs4 for pod openshift-network-diagnostics/network-check-target-ssk4q: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:02:22.251903 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.251864    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4 podName:188765a8-5bc2-48c7-b6ca-c63aa8e4adbf nodeName:}" failed. No retries permitted until 2026-09-07 21:02:54.251846969 +0000 UTC m=+66.333374304 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9ccs4" (UniqueName: "kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4") pod "network-check-target-ssk4q" (UID: "188765a8-5bc2-48c7-b6ca-c63aa8e4adbf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 21:02:22.438727 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.438692    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ggj9j"]
Sep 07 21:02:22.443509 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:02:22.443476    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54304c6d_a18b_46f3_929b_58f0acfe7751.slice/crio-c832cdddb697a87840afb3e295421989a0f524b60318597201951b6d3bb38c52 WatchSource:0}: Error finding container c832cdddb697a87840afb3e295421989a0f524b60318597201951b6d3bb38c52: Status 404 returned error can't find the container with id c832cdddb697a87840afb3e295421989a0f524b60318597201951b6d3bb38c52
Sep 07 21:02:22.554331 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.554300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:22.554480 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.554405    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 21:02:22.554480 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:22.554457    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls podName:917948a6-64e5-49c9-9bc9-31f75632da0b nodeName:}" failed. No retries permitted until 2026-09-07 21:02:23.554442818 +0000 UTC m=+35.635970136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls") pod "dns-default-z6twk" (UID: "917948a6-64e5-49c9-9bc9-31f75632da0b") : secret "dns-default-metrics-tls" not found
Sep 07 21:02:22.563096 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.563066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggj9j" event={"ID":"54304c6d-a18b-46f3-929b-58f0acfe7751","Type":"ContainerStarted","Data":"c832cdddb697a87840afb3e295421989a0f524b60318597201951b6d3bb38c52"}
Sep 07 21:02:22.565607 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.565579    2565 generic.go:356] "Generic (PLEG): container finished" podID="04e701e8-8302-4722-865f-c23db8b82045" containerID="283b7b5fdf8968ab652da1295b0b54eb1bf4a24baa489de4a1df9baa1d4c699d" exitCode=0
Sep 07 21:02:22.565696 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:22.565629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hds9" event={"ID":"04e701e8-8302-4722-865f-c23db8b82045","Type":"ContainerDied","Data":"283b7b5fdf8968ab652da1295b0b54eb1bf4a24baa489de4a1df9baa1d4c699d"}
Sep 07 21:02:23.357804 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.357744    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:23.357996 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.357747    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:23.357996 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.357765    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:23.361388 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.361365    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 21:02:23.361388 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.361390    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 21:02:23.361598 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.361462    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 21:02:23.361598 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.361365    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9r8wd\""
Sep 07 21:02:23.361598 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.361466    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 21:02:23.361747 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.361607    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xs9mt\""
Sep 07 21:02:23.561671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.561638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:23.562206 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:23.561777    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 21:02:23.562206 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:23.561858    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls podName:917948a6-64e5-49c9-9bc9-31f75632da0b nodeName:}" failed. No retries permitted until 2026-09-07 21:02:25.561836463 +0000 UTC m=+37.643363807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls") pod "dns-default-z6twk" (UID: "917948a6-64e5-49c9-9bc9-31f75632da0b") : secret "dns-default-metrics-tls" not found
Sep 07 21:02:23.572554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.572526    2565 generic.go:356] "Generic (PLEG): container finished" podID="04e701e8-8302-4722-865f-c23db8b82045" containerID="689d72c1fa2165e7ded1a830551103ff73da9e2ddf6e28293933920eb59434f1" exitCode=0
Sep 07 21:02:23.572682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:23.572583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hds9" event={"ID":"04e701e8-8302-4722-865f-c23db8b82045","Type":"ContainerDied","Data":"689d72c1fa2165e7ded1a830551103ff73da9e2ddf6e28293933920eb59434f1"}
Sep 07 21:02:24.577937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:24.577902    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-6hds9" event={"ID":"04e701e8-8302-4722-865f-c23db8b82045","Type":"ContainerStarted","Data":"88bb4f20fb99c5d280bda3bda77cdfee4401369e538833a049be4acb52e35b0e"}
Sep 07 21:02:24.600641 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:24.600597    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-6hds9" podStartSLOduration=8.399343573 podStartE2EDuration="36.600583766s" totalImagesPullingTime="28.201240193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:24.599427638 +0000 UTC m=+36.680954977" watchObservedRunningTime="2026-09-07 21:02:24.600583766 +0000 UTC m=+36.682111107"
Sep 07 21:02:25.577818 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:25.577657    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:25.577927 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:25.577803    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 21:02:25.577927 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:25.577886    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls podName:917948a6-64e5-49c9-9bc9-31f75632da0b nodeName:}" failed. No retries permitted until 2026-09-07 21:02:29.577872776 +0000 UTC m=+41.659400095 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls") pod "dns-default-z6twk" (UID: "917948a6-64e5-49c9-9bc9-31f75632da0b") : secret "dns-default-metrics-tls" not found
Sep 07 21:02:25.581115 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:25.581089    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ggj9j" event={"ID":"54304c6d-a18b-46f3-929b-58f0acfe7751","Type":"ContainerStarted","Data":"a83e9bb25dbb8ee88cbc6ebb877692986eabe5022514136d05486e1a1330bf39"}
Sep 07 21:02:25.595487 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:25.595449    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ggj9j" podStartSLOduration=2.194214059 podStartE2EDuration="4.595435383s" totalImagesPullingTime="2.401221324s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:25.59509618 +0000 UTC m=+37.676623525" watchObservedRunningTime="2026-09-07 21:02:25.595435383 +0000 UTC m=+37.676962723"
Sep 07 21:02:29.602435 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:29.602404    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:29.602800 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:29.602508    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 21:02:29.602800 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:29.602563    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls podName:917948a6-64e5-49c9-9bc9-31f75632da0b nodeName:}" failed. No retries permitted until 2026-09-07 21:02:37.602550288 +0000 UTC m=+49.684077607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls") pod "dns-default-z6twk" (UID: "917948a6-64e5-49c9-9bc9-31f75632da0b") : secret "dns-default-metrics-tls" not found
Sep 07 21:02:37.656108 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:37.656075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:37.656522 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:37.656174    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 21:02:37.656522 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:37.656225    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls podName:917948a6-64e5-49c9-9bc9-31f75632da0b nodeName:}" failed. No retries permitted until 2026-09-07 21:02:53.656211315 +0000 UTC m=+65.737738635 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls") pod "dns-default-z6twk" (UID: "917948a6-64e5-49c9-9bc9-31f75632da0b") : secret "dns-default-metrics-tls" not found
Sep 07 21:02:40.395161 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.395125    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qb7kv"]
Sep 07 21:02:40.398065 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.398044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.400643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.400616    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 21:02:40.400743 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.400652    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 21:02:40.404529 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.404172    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 07 21:02:40.404529 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.404416    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 07 21:02:40.404850 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.404829    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6nkm7\""
Sep 07 21:02:40.410017 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.409996    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 07 21:02:40.414691 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.414672    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qb7kv"]
Sep 07 21:02:40.474275 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.474242    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x85fw"]
Sep 07 21:02:40.477037 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.477016    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6545c9995b-hxggn"]
Sep 07 21:02:40.477172 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.477137    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:40.479531 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.479510    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 07 21:02:40.479634 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.479511    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 07 21:02:40.479634 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.479576    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nclq8\""
Sep 07 21:02:40.479731 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.479652    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.481750 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.481732    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 07 21:02:40.482189 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.482169    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 07 21:02:40.482289 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.482212    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 21:02:40.482359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.482341    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 21:02:40.482567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.482553    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 07 21:02:40.482644 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.482555    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5lhcd\""
Sep 07 21:02:40.482779 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.482766    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 07 21:02:40.491890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.491870    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x85fw"]
Sep 07 21:02:40.499582 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.499561    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6545c9995b-hxggn"]
Sep 07 21:02:40.574507 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.574470    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"]
Sep 07 21:02:40.574682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.574543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b1579c-00e4-4922-8bd5-87f58f77c496-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.574682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.574577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfkc\" (UniqueName: \"kubernetes.io/projected/f51d6348-378e-49d1-8c95-65c42db5d93a-kube-api-access-tpfkc\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.574682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.574610    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b1579c-00e4-4922-8bd5-87f58f77c496-tmp\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.574682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.574648    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c9b1579c-00e4-4922-8bd5-87f58f77c496-snapshots\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.574857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.574687    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-default-certificate\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.574857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.574711    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-nginx-conf\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:40.574857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.574752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.575067 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.575029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-stats-auth\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.575201 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.575178    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b1579c-00e4-4922-8bd5-87f58f77c496-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.575282 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.575245    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:40.575359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.575343    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b1579c-00e4-4922-8bd5-87f58f77c496-serving-cert\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.575420 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.575379    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86sf9\" (UniqueName: \"kubernetes.io/projected/c9b1579c-00e4-4922-8bd5-87f58f77c496-kube-api-access-86sf9\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.575613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.575445    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.578713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.578690    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:40.581152 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.581132    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7x4f2\""
Sep 07 21:02:40.581261 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.581238    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 07 21:02:40.581479 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.581465    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 07 21:02:40.581694 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.581678    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 07 21:02:40.591504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.591485    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"]
Sep 07 21:02:40.676022 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.675943    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b1579c-00e4-4922-8bd5-87f58f77c496-serving-cert\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.676022 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.675995    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86sf9\" (UniqueName: \"kubernetes.io/projected/c9b1579c-00e4-4922-8bd5-87f58f77c496-kube-api-access-86sf9\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.676197 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.676197 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676082    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:40.676197 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676133    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b1579c-00e4-4922-8bd5-87f58f77c496-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.676197 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:40.676153    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 21:02:40.676385 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfkc\" (UniqueName: \"kubernetes.io/projected/f51d6348-378e-49d1-8c95-65c42db5d93a-kube-api-access-tpfkc\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.676385 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:40.676215    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:41.176196676 +0000 UTC m=+53.257723999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : secret "router-metrics-certs-default" not found
Sep 07 21:02:40.676385 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676275    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b1579c-00e4-4922-8bd5-87f58f77c496-tmp\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.676385 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676327    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c9b1579c-00e4-4922-8bd5-87f58f77c496-snapshots\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.676385 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-default-certificate\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.676385 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676384    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-nginx-conf\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:40.676655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmsww\" (UniqueName: \"kubernetes.io/projected/7c70d1a9-5bde-4348-af5c-e9b76c28c484-kube-api-access-nmsww\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:40.676655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.676655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-stats-auth\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.676655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676515    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b1579c-00e4-4922-8bd5-87f58f77c496-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.676655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676540    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:40.676655 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:40.676652    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 21:02:40.676954 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:40.676717    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert podName:cd8daeeb-181e-4c8b-b97b-0c06a98faaee nodeName:}" failed. No retries permitted until 2026-09-07 21:02:41.176699241 +0000 UTC m=+53.258226570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x85fw" (UID: "cd8daeeb-181e-4c8b-b97b-0c06a98faaee") : secret "networking-console-plugin-cert" not found
Sep 07 21:02:40.676954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.676789    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b1579c-00e4-4922-8bd5-87f58f77c496-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.676954 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:40.676819    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:41.176805261 +0000 UTC m=+53.258332580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : configmap references non-existent config key: service-ca.crt
Sep 07 21:02:40.677124 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.677103    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c9b1579c-00e4-4922-8bd5-87f58f77c496-snapshots\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.677210 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.677182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-nginx-conf\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:40.677299 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.677282    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9b1579c-00e4-4922-8bd5-87f58f77c496-tmp\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.677473 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.677452    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b1579c-00e4-4922-8bd5-87f58f77c496-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.679788 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.679763    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c9b1579c-00e4-4922-8bd5-87f58f77c496-serving-cert\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.680077 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.680062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-default-certificate\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.680154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.680131    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-stats-auth\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.684677 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.684649    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfkc\" (UniqueName: \"kubernetes.io/projected/f51d6348-378e-49d1-8c95-65c42db5d93a-kube-api-access-tpfkc\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:40.684795 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.684778    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86sf9\" (UniqueName: \"kubernetes.io/projected/c9b1579c-00e4-4922-8bd5-87f58f77c496-kube-api-access-86sf9\") pod \"insights-operator-76bcd6ccf5-qb7kv\" (UID: \"c9b1579c-00e4-4922-8bd5-87f58f77c496\") " pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.706909 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.706891    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv"
Sep 07 21:02:40.777375 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.777347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmsww\" (UniqueName: \"kubernetes.io/projected/7c70d1a9-5bde-4348-af5c-e9b76c28c484-kube-api-access-nmsww\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:40.777499 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.777443    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:40.777562 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:40.777551    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 21:02:40.777621 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:40.777612    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls podName:7c70d1a9-5bde-4348-af5c-e9b76c28c484 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:41.277599172 +0000 UTC m=+53.359126490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kr6zf" (UID: "7c70d1a9-5bde-4348-af5c-e9b76c28c484") : secret "samples-operator-tls" not found
Sep 07 21:02:40.787530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.787499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmsww\" (UniqueName: \"kubernetes.io/projected/7c70d1a9-5bde-4348-af5c-e9b76c28c484-kube-api-access-nmsww\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:40.974995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:40.974906    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-qb7kv"]
Sep 07 21:02:40.978239 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:02:40.978213    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b1579c_00e4_4922_8bd5_87f58f77c496.slice/crio-ffe14d91192c66e14d83a273c9ba179df95532f303de726e5d19e83e6485d1e8 WatchSource:0}: Error finding container ffe14d91192c66e14d83a273c9ba179df95532f303de726e5d19e83e6485d1e8: Status 404 returned error can't find the container with id ffe14d91192c66e14d83a273c9ba179df95532f303de726e5d19e83e6485d1e8
Sep 07 21:02:41.179270 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:41.179235    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:41.179451 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:41.179304    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:41.179451 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:41.179326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:41.179451 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:41.179419    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 21:02:41.179578 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:41.179417    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 21:02:41.179578 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:41.179466    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert podName:cd8daeeb-181e-4c8b-b97b-0c06a98faaee nodeName:}" failed. No retries permitted until 2026-09-07 21:02:42.179453146 +0000 UTC m=+54.260980465 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x85fw" (UID: "cd8daeeb-181e-4c8b-b97b-0c06a98faaee") : secret "networking-console-plugin-cert" not found
Sep 07 21:02:41.179578 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:41.179510    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:42.179493001 +0000 UTC m=+54.261020319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : configmap references non-existent config key: service-ca.crt
Sep 07 21:02:41.179578 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:41.179533    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:42.179522512 +0000 UTC m=+54.261049836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : secret "router-metrics-certs-default" not found
Sep 07 21:02:41.280434 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:41.280357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:41.280546 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:41.280500    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 21:02:41.280582 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:41.280561    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls podName:7c70d1a9-5bde-4348-af5c-e9b76c28c484 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:42.280545439 +0000 UTC m=+54.362072758 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kr6zf" (UID: "7c70d1a9-5bde-4348-af5c-e9b76c28c484") : secret "samples-operator-tls" not found
Sep 07 21:02:41.619644 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:41.619607    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv" event={"ID":"c9b1579c-00e4-4922-8bd5-87f58f77c496","Type":"ContainerStarted","Data":"ffe14d91192c66e14d83a273c9ba179df95532f303de726e5d19e83e6485d1e8"}
Sep 07 21:02:42.187818 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:42.187780    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:42.187818 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:42.187824    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:42.188154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:42.187877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:42.188154 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:42.187952    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:44.187935041 +0000 UTC m=+56.269462403 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : configmap references non-existent config key: service-ca.crt
Sep 07 21:02:42.188154 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:42.188009    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 21:02:42.188154 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:42.188049    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:44.188037156 +0000 UTC m=+56.269564474 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : secret "router-metrics-certs-default" not found
Sep 07 21:02:42.188154 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:42.188010    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 21:02:42.188154 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:42.188078    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert podName:cd8daeeb-181e-4c8b-b97b-0c06a98faaee nodeName:}" failed. No retries permitted until 2026-09-07 21:02:44.188071965 +0000 UTC m=+56.269599284 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x85fw" (UID: "cd8daeeb-181e-4c8b-b97b-0c06a98faaee") : secret "networking-console-plugin-cert" not found
Sep 07 21:02:42.288451 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:42.288415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:42.288611 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:42.288587    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 21:02:42.288689 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:42.288678    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls podName:7c70d1a9-5bde-4348-af5c-e9b76c28c484 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:44.288657951 +0000 UTC m=+56.370185270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kr6zf" (UID: "7c70d1a9-5bde-4348-af5c-e9b76c28c484") : secret "samples-operator-tls" not found
Sep 07 21:02:43.626604 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.626567    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv" event={"ID":"c9b1579c-00e4-4922-8bd5-87f58f77c496","Type":"ContainerStarted","Data":"adc67c784d2ae1c4c3962ee9f92463b61d18e4035b48102b33248baf9ab3a988"}
Sep 07 21:02:43.659156 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.659100    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv" podStartSLOduration=1.8423946469999999 podStartE2EDuration="3.659081581s" totalImagesPullingTime="1.816686934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:43.65837852 +0000 UTC m=+55.739905855" watchObservedRunningTime="2026-09-07 21:02:43.659081581 +0000 UTC m=+55.740608922"
Sep 07 21:02:43.717576 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.717539    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fq5qz"]
Sep 07 21:02:43.720737 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.720704    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:43.723372 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.723341    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 21:02:43.723501 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.723351    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 21:02:43.723501 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.723382    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvs4x\""
Sep 07 21:02:43.736525 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.736499    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fq5qz"]
Sep 07 21:02:43.902552 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.902467    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f9250458-8d8f-42c7-bc7e-0dfca2791158-crio-socket\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:43.902552 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.902515    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9250458-8d8f-42c7-bc7e-0dfca2791158-data-volume\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:43.902552 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.902533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdb8\" (UniqueName: \"kubernetes.io/projected/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-api-access-nhdb8\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:43.902757 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.902555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:43.902757 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:43.902652    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.003777 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.003733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f9250458-8d8f-42c7-bc7e-0dfca2791158-crio-socket\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.003777 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.003781    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9250458-8d8f-42c7-bc7e-0dfca2791158-data-volume\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.003992 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.003799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdb8\" (UniqueName: \"kubernetes.io/projected/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-api-access-nhdb8\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.003992 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.003863    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f9250458-8d8f-42c7-bc7e-0dfca2791158-crio-socket\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.003992 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.003917    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.004097 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.004016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.004097 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.004055    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 21:02:44.004163 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.004134    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:44.504111971 +0000 UTC m=+56.585639303 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : configmap "kube-rbac-proxy" not found
Sep 07 21:02:44.004163 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.004137    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:44.004163 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.004152    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f9250458-8d8f-42c7-bc7e-0dfca2791158-data-volume\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.004265 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.004192    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:44.504177005 +0000 UTC m=+56.585704333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:44.013492 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.013467    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdb8\" (UniqueName: \"kubernetes.io/projected/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-api-access-nhdb8\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.205648 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.205553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:44.205648 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.205600    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:44.205887 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.205711    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 21:02:44.205887 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.205763    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:48.205740769 +0000 UTC m=+60.287268106 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : configmap references non-existent config key: service-ca.crt
Sep 07 21:02:44.205887 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.205791    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert podName:cd8daeeb-181e-4c8b-b97b-0c06a98faaee nodeName:}" failed. No retries permitted until 2026-09-07 21:02:48.205782702 +0000 UTC m=+60.287310025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x85fw" (UID: "cd8daeeb-181e-4c8b-b97b-0c06a98faaee") : secret "networking-console-plugin-cert" not found
Sep 07 21:02:44.205887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.205830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:44.206106 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.205990    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 21:02:44.206106 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.206037    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:48.206023323 +0000 UTC m=+60.287550655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : secret "router-metrics-certs-default" not found
Sep 07 21:02:44.307267 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.307229    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:44.307424 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.307381    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 21:02:44.307466 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.307451    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls podName:7c70d1a9-5bde-4348-af5c-e9b76c28c484 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:48.30743617 +0000 UTC m=+60.388963490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kr6zf" (UID: "7c70d1a9-5bde-4348-af5c-e9b76c28c484") : secret "samples-operator-tls" not found
Sep 07 21:02:44.508683 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.508603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.508844 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:44.508743    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:44.508844 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.508754    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:44.508844 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.508816    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:45.508800633 +0000 UTC m=+57.590327952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:44.508844 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.508838    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 21:02:44.509023 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:44.508892    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:45.508876483 +0000 UTC m=+57.590403823 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : configmap "kube-rbac-proxy" not found
Sep 07 21:02:45.448236 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:45.448207    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-22b4b_57134fda-ea48-44a8-920e-b435bb868eb4/dns-node-resolver/0.log"
Sep 07 21:02:45.519742 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:45.519707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:45.519889 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:45.519776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:45.519889 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:45.519831    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 21:02:45.519889 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:45.519871    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:45.519889 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:45.519882    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:47.519868403 +0000 UTC m=+59.601395745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : configmap "kube-rbac-proxy" not found
Sep 07 21:02:45.520061 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:45.519944    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:47.519929827 +0000 UTC m=+59.601457172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:46.645691 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:46.645662    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h94gv_02eff29a-6dd4-495c-84b1-97c952ad2b0b/node-ca/0.log"
Sep 07 21:02:47.246111 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:47.246067    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggj9j_54304c6d-a18b-46f3-929b-58f0acfe7751/serve-healthcheck-canary/0.log"
Sep 07 21:02:47.536216 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:47.536037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:47.536216 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:47.536124    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:47.536458 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:47.536272    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 21:02:47.536458 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:47.536334    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:47.536458 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:47.536350    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:51.536329674 +0000 UTC m=+63.617857015 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : configmap "kube-rbac-proxy" not found
Sep 07 21:02:47.536458 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:47.536386    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:51.536370823 +0000 UTC m=+63.617898143 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:47.560037 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:47.560015    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gxfll"
Sep 07 21:02:48.242060 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:48.242027    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:48.242423 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:48.242095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:48.242423 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:48.242117    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:48.242423 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:48.242171    2565 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 07 21:02:48.242423 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:48.242198    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 21:02:48.242423 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:48.242233    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:56.242219084 +0000 UTC m=+68.323746403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : secret "router-metrics-certs-default" not found
Sep 07 21:02:48.242423 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:48.242246    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert podName:cd8daeeb-181e-4c8b-b97b-0c06a98faaee nodeName:}" failed. No retries permitted until 2026-09-07 21:02:56.242239953 +0000 UTC m=+68.323767272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x85fw" (UID: "cd8daeeb-181e-4c8b-b97b-0c06a98faaee") : secret "networking-console-plugin-cert" not found
Sep 07 21:02:48.242423 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:48.242256    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle podName:f51d6348-378e-49d1-8c95-65c42db5d93a nodeName:}" failed. No retries permitted until 2026-09-07 21:02:56.242250743 +0000 UTC m=+68.323778062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle") pod "router-default-6545c9995b-hxggn" (UID: "f51d6348-378e-49d1-8c95-65c42db5d93a") : configmap references non-existent config key: service-ca.crt
Sep 07 21:02:48.342692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:48.342663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:48.342873 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:48.342822    2565 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 07 21:02:48.343043 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:48.342889    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls podName:7c70d1a9-5bde-4348-af5c-e9b76c28c484 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:56.342868955 +0000 UTC m=+68.424396302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-kr6zf" (UID: "7c70d1a9-5bde-4348-af5c-e9b76c28c484") : secret "samples-operator-tls" not found
Sep 07 21:02:51.567151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:51.567111    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:51.567567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:51.567223    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:51.567567 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:51.567296    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 21:02:51.567567 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:51.567296    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:51.567567 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:51.567360    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:59.567346731 +0000 UTC m=+71.648874050 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : configmap "kube-rbac-proxy" not found
Sep 07 21:02:51.567567 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:51.567372    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:02:59.567366793 +0000 UTC m=+71.648894112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:53.683057 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:53.683028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:53.685233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:53.685212    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/917948a6-64e5-49c9-9bc9-31f75632da0b-metrics-tls\") pod \"dns-default-z6twk\" (UID: \"917948a6-64e5-49c9-9bc9-31f75632da0b\") " pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:53.919800 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:53.919771    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7njqf\""
Sep 07 21:02:53.927732 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:53.927713    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:54.087458 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.087427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:02:54.090004 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.089986    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 21:02:54.098072 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:54.098056    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 07 21:02:54.098117 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:54.098110    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs podName:2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8 nodeName:}" failed. No retries permitted until 2026-09-07 21:03:58.098090636 +0000 UTC m=+130.179617968 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs") pod "network-metrics-daemon-xbp8h" (UID: "2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8") : secret "metrics-daemon-secret" not found
Sep 07 21:02:54.188801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.188734    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:54.191015 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.190998    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 21:02:54.195769 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.195747    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z6twk"]
Sep 07 21:02:54.199264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:02:54.199231    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod917948a6_64e5_49c9_9bc9_31f75632da0b.slice/crio-d336de3158f358d9b67fcb06dc3adefbb4076bb0092c63e8de8873119bf47eb2 WatchSource:0}: Error finding container d336de3158f358d9b67fcb06dc3adefbb4076bb0092c63e8de8873119bf47eb2: Status 404 returned error can't find the container with id d336de3158f358d9b67fcb06dc3adefbb4076bb0092c63e8de8873119bf47eb2
Sep 07 21:02:54.201281 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.201180    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/12c4533d-f234-4b23-b343-c3cebf3279e9-original-pull-secret\") pod \"global-pull-secret-syncer-9tlvn\" (UID: \"12c4533d-f234-4b23-b343-c3cebf3279e9\") " pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:54.277212 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.277190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9tlvn"
Sep 07 21:02:54.289928 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.289905    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:54.292544 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.292528    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 21:02:54.303583 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.303558    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 21:02:54.313985 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.313945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ccs4\" (UniqueName: \"kubernetes.io/projected/188765a8-5bc2-48c7-b6ca-c63aa8e4adbf-kube-api-access-9ccs4\") pod \"network-check-target-ssk4q\" (UID: \"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf\") " pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:54.545919 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.545837    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9tlvn"]
Sep 07 21:02:54.548717 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:02:54.548686    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c4533d_f234_4b23_b343_c3cebf3279e9.slice/crio-a76bc317c4eb6ad8e0b302c17b8969932ba7fe48407837fa4a2736f4ee38916d WatchSource:0}: Error finding container a76bc317c4eb6ad8e0b302c17b8969932ba7fe48407837fa4a2736f4ee38916d: Status 404 returned error can't find the container with id a76bc317c4eb6ad8e0b302c17b8969932ba7fe48407837fa4a2736f4ee38916d
Sep 07 21:02:54.572852 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.572833    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xs9mt\""
Sep 07 21:02:54.581058 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.581043    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:54.654403 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.654376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9tlvn" event={"ID":"12c4533d-f234-4b23-b343-c3cebf3279e9","Type":"ContainerStarted","Data":"a76bc317c4eb6ad8e0b302c17b8969932ba7fe48407837fa4a2736f4ee38916d"}
Sep 07 21:02:54.655468 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.655449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6twk" event={"ID":"917948a6-64e5-49c9-9bc9-31f75632da0b","Type":"ContainerStarted","Data":"d336de3158f358d9b67fcb06dc3adefbb4076bb0092c63e8de8873119bf47eb2"}
Sep 07 21:02:54.850798 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:54.850768    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ssk4q"]
Sep 07 21:02:54.853707 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:02:54.853683    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod188765a8_5bc2_48c7_b6ca_c63aa8e4adbf.slice/crio-539cf6095407fd50ad19e7705f8ab1d93c3b5a245b67cab9a8f4da3fd40e23df WatchSource:0}: Error finding container 539cf6095407fd50ad19e7705f8ab1d93c3b5a245b67cab9a8f4da3fd40e23df: Status 404 returned error can't find the container with id 539cf6095407fd50ad19e7705f8ab1d93c3b5a245b67cab9a8f4da3fd40e23df
Sep 07 21:02:55.660278 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:55.660234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ssk4q" event={"ID":"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf","Type":"ContainerStarted","Data":"539cf6095407fd50ad19e7705f8ab1d93c3b5a245b67cab9a8f4da3fd40e23df"}
Sep 07 21:02:56.308403 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.308355    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:56.308859 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.308469    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:56.308859 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.308502    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:02:56.308859 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:56.308632    2565 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 07 21:02:56.308859 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:56.308702    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert podName:cd8daeeb-181e-4c8b-b97b-0c06a98faaee nodeName:}" failed. No retries permitted until 2026-09-07 21:03:12.308681329 +0000 UTC m=+84.390208673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-x85fw" (UID: "cd8daeeb-181e-4c8b-b97b-0c06a98faaee") : secret "networking-console-plugin-cert" not found
Sep 07 21:02:56.309590 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.309542    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f51d6348-378e-49d1-8c95-65c42db5d93a-service-ca-bundle\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:56.313214 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.313148    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f51d6348-378e-49d1-8c95-65c42db5d93a-metrics-certs\") pod \"router-default-6545c9995b-hxggn\" (UID: \"f51d6348-378e-49d1-8c95-65c42db5d93a\") " pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:56.393389 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.393358    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-5lhcd\""
Sep 07 21:02:56.401688 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.401517    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:56.409739 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.409702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:56.413228 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.413203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7c70d1a9-5bde-4348-af5c-e9b76c28c484-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-kr6zf\" (UID: \"7c70d1a9-5bde-4348-af5c-e9b76c28c484\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:56.490558 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.490493    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7x4f2\""
Sep 07 21:02:56.499037 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.498920    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"
Sep 07 21:02:56.667709 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.667440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6twk" event={"ID":"917948a6-64e5-49c9-9bc9-31f75632da0b","Type":"ContainerStarted","Data":"5e0510e97a53770680de90129a8203f0cb3385ce520c34f709a52facfce79f7b"}
Sep 07 21:02:56.667709 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.667501    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z6twk" event={"ID":"917948a6-64e5-49c9-9bc9-31f75632da0b","Type":"ContainerStarted","Data":"b4ba84cb4859dbc9441454240102febd78a20b8309da8663a46d0f02f857e0fb"}
Sep 07 21:02:56.667709 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.667705    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z6twk"
Sep 07 21:02:56.686671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.686612    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z6twk" podStartSLOduration=33.493837387 podStartE2EDuration="35.686592248s" totalImagesPullingTime="2.192754861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:02:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:56.68571293 +0000 UTC m=+68.767240273" watchObservedRunningTime="2026-09-07 21:02:56.686592248 +0000 UTC m=+68.768119641"
Sep 07 21:02:56.693440 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.693414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6545c9995b-hxggn"]
Sep 07 21:02:56.697083 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:02:56.697057    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51d6348_378e_49d1_8c95_65c42db5d93a.slice/crio-81a991bfe747041229bdf0687039ba5c64d01194191f81e3760704b00df1cc54 WatchSource:0}: Error finding container 81a991bfe747041229bdf0687039ba5c64d01194191f81e3760704b00df1cc54: Status 404 returned error can't find the container with id 81a991bfe747041229bdf0687039ba5c64d01194191f81e3760704b00df1cc54
Sep 07 21:02:56.791887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:56.791749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf"]
Sep 07 21:02:57.671374 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:57.671330    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf" event={"ID":"7c70d1a9-5bde-4348-af5c-e9b76c28c484","Type":"ContainerStarted","Data":"cb9a42e0aed11653047843c1af080b7ccac5c406e753cdfc0e6cc5f978c1deaa"}
Sep 07 21:02:57.673104 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:57.673079    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6545c9995b-hxggn" event={"ID":"f51d6348-378e-49d1-8c95-65c42db5d93a","Type":"ContainerStarted","Data":"9c1b5f12004a54dcb5a168e0afa2a90db6faad57fb2f1c8056c9c0cfc81c17bd"}
Sep 07 21:02:57.673238 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:57.673131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6545c9995b-hxggn" event={"ID":"f51d6348-378e-49d1-8c95-65c42db5d93a","Type":"ContainerStarted","Data":"81a991bfe747041229bdf0687039ba5c64d01194191f81e3760704b00df1cc54"}
Sep 07 21:02:57.692552 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:57.692496    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6545c9995b-hxggn" podStartSLOduration=17.692476076 podStartE2EDuration="17.692476076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:02:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:57.691826359 +0000 UTC m=+69.773353704" watchObservedRunningTime="2026-09-07 21:02:57.692476076 +0000 UTC m=+69.774003419"
Sep 07 21:02:58.402386 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:58.402345    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:58.405340 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:58.405311    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:58.675934 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:58.675842    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:58.677327 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:58.677303    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6545c9995b-hxggn"
Sep 07 21:02:59.643077 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:59.643037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:59.643263 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:59.643097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:59.643263 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:59.643225    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:59.643372 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:02:59.643297    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls podName:f9250458-8d8f-42c7-bc7e-0dfca2791158 nodeName:}" failed. No retries permitted until 2026-09-07 21:03:15.643279198 +0000 UTC m=+87.724806517 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fq5qz" (UID: "f9250458-8d8f-42c7-bc7e-0dfca2791158") : secret "insights-runtime-extractor-tls" not found
Sep 07 21:02:59.643647 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:59.643616    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f9250458-8d8f-42c7-bc7e-0dfca2791158-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:02:59.680645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:59.680596    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ssk4q" event={"ID":"188765a8-5bc2-48c7-b6ca-c63aa8e4adbf","Type":"ContainerStarted","Data":"a9ad58c833a5cc685d86333ccb52900c0b943fdb6c627e580628a9dd44986bb0"}
Sep 07 21:02:59.681155 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:59.680717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:02:59.682633 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:59.682601    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9tlvn" event={"ID":"12c4533d-f234-4b23-b343-c3cebf3279e9","Type":"ContainerStarted","Data":"53983eac9aa29f2a0d3c168f9680c3dfc19565c2c2d89b6819d26ea4e9923a0c"}
Sep 07 21:02:59.697405 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:59.697325    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ssk4q" podStartSLOduration=67.058764101 podStartE2EDuration="1m11.69731016s" totalImagesPullingTime="4.638546059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:59.696571469 +0000 UTC m=+71.778098812" watchObservedRunningTime="2026-09-07 21:02:59.69731016 +0000 UTC m=+71.778837497"
Sep 07 21:02:59.711358 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:02:59.711298    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-9tlvn" podStartSLOduration=66.761811059 podStartE2EDuration="1m11.711280927s" totalImagesPullingTime="4.949469868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:02:59.710794505 +0000 UTC m=+71.792321850" watchObservedRunningTime="2026-09-07 21:02:59.711280927 +0000 UTC m=+71.792808269"
Sep 07 21:03:01.692447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:01.692410    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf" event={"ID":"7c70d1a9-5bde-4348-af5c-e9b76c28c484","Type":"ContainerStarted","Data":"5fa072adfebae258dc8dface22532ce891e61d11b925ece1115b1dcdc5eba370"}
Sep 07 21:03:01.692890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:01.692471    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf" event={"ID":"7c70d1a9-5bde-4348-af5c-e9b76c28c484","Type":"ContainerStarted","Data":"744957afdc0784b4a9eb68300a94645c86ee8674edc3224ce2592fece572418b"}
Sep 07 21:03:01.709224 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:01.709178    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-kr6zf" podStartSLOduration=17.907919354 podStartE2EDuration="21.709163015s" totalImagesPullingTime="3.801243661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:01.707535484 +0000 UTC m=+73.789062824" watchObservedRunningTime="2026-09-07 21:03:01.709163015 +0000 UTC m=+73.790690355"
Sep 07 21:03:06.674954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:06.674923    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z6twk"
Sep 07 21:03:08.050758 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.050727    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"]
Sep 07 21:03:08.054000 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.053977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.056478 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.056426    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 07 21:03:08.056600 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.056572    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 07 21:03:08.056806 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.056783    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 21:03:08.057364 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.057345    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 21:03:08.057541 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.057360    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 07 21:03:08.057616 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.057540    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 21:03:08.057616 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.057432    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 07 21:03:08.087771 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.087747    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"]
Sep 07 21:03:08.104709 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.104683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-ca\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.104851 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.104717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.104851 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.104764    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.104851 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.104797    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8lc\" (UniqueName: \"kubernetes.io/projected/58571ca7-55e0-4ad1-9bee-13ef687333ba-kube-api-access-jj8lc\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.105034 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.104875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/58571ca7-55e0-4ad1-9bee-13ef687333ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.105034 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.104926    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-hub\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.206132 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.206095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8lc\" (UniqueName: \"kubernetes.io/projected/58571ca7-55e0-4ad1-9bee-13ef687333ba-kube-api-access-jj8lc\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.206132 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.206142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/58571ca7-55e0-4ad1-9bee-13ef687333ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.206376 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.206172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-hub\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.206376 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.206224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-ca\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.206376 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.206239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.206376 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.206253    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.207033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.207007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/58571ca7-55e0-4ad1-9bee-13ef687333ba-additional-service-ca\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.208880 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.208853    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-ca\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.209016 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.208919    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.209226 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.209202    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.209271 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.209229    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/58571ca7-55e0-4ad1-9bee-13ef687333ba-hub\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.228046 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.228020    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8lc\" (UniqueName: \"kubernetes.io/projected/58571ca7-55e0-4ad1-9bee-13ef687333ba-kube-api-access-jj8lc\") pod \"cluster-proxy-proxy-agent-64857fd657-8jbr8\" (UID: \"58571ca7-55e0-4ad1-9bee-13ef687333ba\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.379735 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.379702    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"
Sep 07 21:03:08.655504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.655427    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8"]
Sep 07 21:03:08.658395 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:08.658370    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58571ca7_55e0_4ad1_9bee_13ef687333ba.slice/crio-cdf8aab55b7b920ee1205a12c0aed1717c9855ab22b4adf9cb1ce4efe5442bf1 WatchSource:0}: Error finding container cdf8aab55b7b920ee1205a12c0aed1717c9855ab22b4adf9cb1ce4efe5442bf1: Status 404 returned error can't find the container with id cdf8aab55b7b920ee1205a12c0aed1717c9855ab22b4adf9cb1ce4efe5442bf1
Sep 07 21:03:08.718216 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:08.718185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8" event={"ID":"58571ca7-55e0-4ad1-9bee-13ef687333ba","Type":"ContainerStarted","Data":"cdf8aab55b7b920ee1205a12c0aed1717c9855ab22b4adf9cb1ce4efe5442bf1"}
Sep 07 21:03:12.342159 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.342134    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:03:12.344440 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.344410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/cd8daeeb-181e-4c8b-b97b-0c06a98faaee-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-x85fw\" (UID: \"cd8daeeb-181e-4c8b-b97b-0c06a98faaee\") " pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:03:12.589298 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.589271    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nclq8\""
Sep 07 21:03:12.597346 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.597325    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw"
Sep 07 21:03:12.734690 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.734651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8" event={"ID":"58571ca7-55e0-4ad1-9bee-13ef687333ba","Type":"ContainerStarted","Data":"f24088be5997a98c6c0adbe3ff224d18e191a84a84486ea2fb327ca2b2d01f0c"}
Sep 07 21:03:12.734690 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.734693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8" event={"ID":"58571ca7-55e0-4ad1-9bee-13ef687333ba","Type":"ContainerStarted","Data":"7a8a52175e5977e8790a9e10e5031c417511419a6a62c544f21b097d65cff8bb"}
Sep 07 21:03:12.734883 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.734706    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8" event={"ID":"58571ca7-55e0-4ad1-9bee-13ef687333ba","Type":"ContainerStarted","Data":"505ff3fe86cd14c1ce65c98b4f892dfe5bf25ff3b77d6463c74b545eac73fbe6"}
Sep 07 21:03:12.755854 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.753631    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-64857fd657-8jbr8" podStartSLOduration=1.20350872 podStartE2EDuration="4.753609135s" totalImagesPullingTime="3.550100415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:12.752808683 +0000 UTC m=+84.834336025" watchObservedRunningTime="2026-09-07 21:03:12.753609135 +0000 UTC m=+84.835136484"
Sep 07 21:03:12.866164 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:12.866085    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-x85fw"]
Sep 07 21:03:12.868997 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:12.868953    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8daeeb_181e_4c8b_b97b_0c06a98faaee.slice/crio-f0a48916822077f21e99137a0eb86c43b0b30d750bfb319609ce597ad252f369 WatchSource:0}: Error finding container f0a48916822077f21e99137a0eb86c43b0b30d750bfb319609ce597ad252f369: Status 404 returned error can't find the container with id f0a48916822077f21e99137a0eb86c43b0b30d750bfb319609ce597ad252f369
Sep 07 21:03:13.739850 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:13.739812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw" event={"ID":"cd8daeeb-181e-4c8b-b97b-0c06a98faaee","Type":"ContainerStarted","Data":"f0a48916822077f21e99137a0eb86c43b0b30d750bfb319609ce597ad252f369"}
Sep 07 21:03:14.744030 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:14.743997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw" event={"ID":"cd8daeeb-181e-4c8b-b97b-0c06a98faaee","Type":"ContainerStarted","Data":"f9bb0335084d102cecac9b17c9d1e80b80cfbdfab5a8ee6bb688b425fb0a120c"}
Sep 07 21:03:14.759880 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:14.759830    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-x85fw" podStartSLOduration=33.552701549 podStartE2EDuration="34.759812836s" totalImagesPullingTime="1.207111287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:14.759538595 +0000 UTC m=+86.841065937" watchObservedRunningTime="2026-09-07 21:03:14.759812836 +0000 UTC m=+86.841340176"
Sep 07 21:03:15.669034 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:15.669004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:03:15.671203 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:15.671176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f9250458-8d8f-42c7-bc7e-0dfca2791158-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fq5qz\" (UID: \"f9250458-8d8f-42c7-bc7e-0dfca2791158\") " pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:03:15.831975 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:15.831944    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vvs4x\""
Sep 07 21:03:15.839791 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:15.839757    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fq5qz"
Sep 07 21:03:16.107692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:16.107665    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fq5qz"]
Sep 07 21:03:16.110799 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:16.110764    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9250458_8d8f_42c7_bc7e_0dfca2791158.slice/crio-727034f64c8add8d2a28254ff7ac24945f83aff489b8a417e5446eb0ef22f5c7 WatchSource:0}: Error finding container 727034f64c8add8d2a28254ff7ac24945f83aff489b8a417e5446eb0ef22f5c7: Status 404 returned error can't find the container with id 727034f64c8add8d2a28254ff7ac24945f83aff489b8a417e5446eb0ef22f5c7
Sep 07 21:03:16.753748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:16.753708    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fq5qz" event={"ID":"f9250458-8d8f-42c7-bc7e-0dfca2791158","Type":"ContainerStarted","Data":"90d2e6913b2324a2a191e2a9706c29c4aa5ec77601599a519458249ab83b60cf"}
Sep 07 21:03:16.753909 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:16.753816    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fq5qz" event={"ID":"f9250458-8d8f-42c7-bc7e-0dfca2791158","Type":"ContainerStarted","Data":"727034f64c8add8d2a28254ff7ac24945f83aff489b8a417e5446eb0ef22f5c7"}
Sep 07 21:03:17.759476 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:17.759437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fq5qz" event={"ID":"f9250458-8d8f-42c7-bc7e-0dfca2791158","Type":"ContainerStarted","Data":"af3a2cc02e2429c453b0154ea60e90ccc21de5d2350ac5c23945e5cfbf9f1926"}
Sep 07 21:03:18.764408 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:18.764306    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fq5qz" event={"ID":"f9250458-8d8f-42c7-bc7e-0dfca2791158","Type":"ContainerStarted","Data":"c44ca5c0061d01a2f021d6042eef6cf0339f203252ee56f6db07275f504f2818"}
Sep 07 21:03:18.781103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:18.781058    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fq5qz" podStartSLOduration=33.532614966 podStartE2EDuration="35.781043306s" totalImagesPullingTime="2.24842834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:02:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:18.780128751 +0000 UTC m=+90.861656091" watchObservedRunningTime="2026-09-07 21:03:18.781043306 +0000 UTC m=+90.862570637"
Sep 07 21:03:24.495094 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.495065    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sxc5f"]
Sep 07 21:03:24.499227 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.499210    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.501811 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.501786    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4kr28\""
Sep 07 21:03:24.501976 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.501867    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 21:03:24.501976 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.501887    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 21:03:24.502557 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.502539    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 21:03:24.502670 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.502538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 21:03:24.502670 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.502575    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 21:03:24.502670 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.502563    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 21:03:24.520878 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.520855    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"]
Sep 07 21:03:24.524301 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.524285    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.526804 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.526782    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 07 21:03:24.526895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.526799    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 07 21:03:24.526895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.526801    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tjt6g\""
Sep 07 21:03:24.527148 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.527132    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 07 21:03:24.537441 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.537422    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"]
Sep 07 21:03:24.638418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638384    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.638418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxrd\" (UniqueName: \"kubernetes.io/projected/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-api-access-wkxrd\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.638701 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638453    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa52f873-c713-46b2-b054-0bce67a3111a-metrics-client-ca\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.638701 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638504    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-sys\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.638701 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638540    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-wtmp\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.638701 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.638701 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638686    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-textfile\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.638929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.638929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638755    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.638929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638800    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.638929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxgw\" (UniqueName: \"kubernetes.io/projected/aa52f873-c713-46b2-b054-0bce67a3111a-kube-api-access-dsxgw\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.638929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638886    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-tls\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.638929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638902    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.638929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-root\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.639178 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.638949    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.739949 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.739915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa52f873-c713-46b2-b054-0bce67a3111a-metrics-client-ca\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.739949 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.739952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-sys\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-sys\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-wtmp\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.740177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740163    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-textfile\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.740383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740199    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.740383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740234    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.740383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740245    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-wtmp\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740383 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:03:24.740332    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 07 21:03:24.740603 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:03:24.740409    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-tls podName:8ef9a728-9e2b-4341-8554-7bcabaecdf7b nodeName:}" failed. No retries permitted until 2026-09-07 21:03:25.240389001 +0000 UTC m=+97.321916322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-tls") pod "kube-state-metrics-74cb57df7-nspqx" (UID: "8ef9a728-9e2b-4341-8554-7bcabaecdf7b") : secret "kube-state-metrics-tls" not found
Sep 07 21:03:24.740603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740458    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-textfile\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxgw\" (UniqueName: \"kubernetes.io/projected/aa52f873-c713-46b2-b054-0bce67a3111a-kube-api-access-dsxgw\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740537    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa52f873-c713-46b2-b054-0bce67a3111a-metrics-client-ca\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-tls\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740586    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-root\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.740952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740673    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.740952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740696    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxrd\" (UniqueName: \"kubernetes.io/projected/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-api-access-wkxrd\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.741180 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.740990    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-metrics-client-ca\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.741180 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.741051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/aa52f873-c713-46b2-b054-0bce67a3111a-root\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.741283 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.741195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-accelerators-collector-config\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.741441 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.741411    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-volume-directive-shadow\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.741759 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.741733    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.742819 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.742791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-tls\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.742909 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.742840    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.743159 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.743141    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa52f873-c713-46b2-b054-0bce67a3111a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.753358 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.753310    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxgw\" (UniqueName: \"kubernetes.io/projected/aa52f873-c713-46b2-b054-0bce67a3111a-kube-api-access-dsxgw\") pod \"node-exporter-sxc5f\" (UID: \"aa52f873-c713-46b2-b054-0bce67a3111a\") " pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.753556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.753539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxrd\" (UniqueName: \"kubernetes.io/projected/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-api-access-wkxrd\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:24.808678 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:24.808648    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sxc5f"
Sep 07 21:03:24.821197 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:24.821167    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa52f873_c713_46b2_b054_0bce67a3111a.slice/crio-b3a0ef92b1e8775a1932a384e7cd8f3f76ec3d5eb9a398ae6c526e78f4bde506 WatchSource:0}: Error finding container b3a0ef92b1e8775a1932a384e7cd8f3f76ec3d5eb9a398ae6c526e78f4bde506: Status 404 returned error can't find the container with id b3a0ef92b1e8775a1932a384e7cd8f3f76ec3d5eb9a398ae6c526e78f4bde506
Sep 07 21:03:25.245048 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:25.245007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:25.247383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:25.247362    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8ef9a728-9e2b-4341-8554-7bcabaecdf7b-kube-state-metrics-tls\") pod \"kube-state-metrics-74cb57df7-nspqx\" (UID: \"8ef9a728-9e2b-4341-8554-7bcabaecdf7b\") " pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:25.432812 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:25.432784    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"
Sep 07 21:03:25.715056 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:25.713994    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-74cb57df7-nspqx"]
Sep 07 21:03:25.789053 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:25.789000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxc5f" event={"ID":"aa52f873-c713-46b2-b054-0bce67a3111a","Type":"ContainerStarted","Data":"b3a0ef92b1e8775a1932a384e7cd8f3f76ec3d5eb9a398ae6c526e78f4bde506"}
Sep 07 21:03:25.843864 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:25.843836    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef9a728_9e2b_4341_8554_7bcabaecdf7b.slice/crio-27b9febb567d424a43a00adc6bb17cd39fcb29c631a3aa0428e6ae60ef8e2ade WatchSource:0}: Error finding container 27b9febb567d424a43a00adc6bb17cd39fcb29c631a3aa0428e6ae60ef8e2ade: Status 404 returned error can't find the container with id 27b9febb567d424a43a00adc6bb17cd39fcb29c631a3aa0428e6ae60ef8e2ade
Sep 07 21:03:26.573286 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.573254    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6c676d7495-4qq7m"]
Sep 07 21:03:26.576926 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.576904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.580323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.579656    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 07 21:03:26.580323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.579660    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-s2nx2\""
Sep 07 21:03:26.580323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.579697    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 07 21:03:26.580323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.579711    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 07 21:03:26.580323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.580028    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 07 21:03:26.580323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.580072    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 07 21:03:26.580323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.580190    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3kka5id44ar9j\""
Sep 07 21:03:26.587634 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.587613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c676d7495-4qq7m"]
Sep 07 21:03:26.657688 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.657649    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.657856 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.657717    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.657856 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.657766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-metrics-client-ca\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.657856 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.657830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.658003 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.657867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-tls\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.658003 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.657888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9bj\" (UniqueName: \"kubernetes.io/projected/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-kube-api-access-df9bj\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.658003 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.657915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.658003 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.657957    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-grpc-tls\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.759149 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.759112    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-grpc-tls\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.759595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.759169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.759595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.759220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.759595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.759249    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-metrics-client-ca\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.759595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.759297    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.759595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.759334    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-tls\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.759595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.759357    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9bj\" (UniqueName: \"kubernetes.io/projected/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-kube-api-access-df9bj\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.759595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.759387    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.760190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.760134    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-metrics-client-ca\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.762497 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.762339    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.762497 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.762438    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.762497 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.762463    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-grpc-tls\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.762846 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.762791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.763750 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.763720    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.764010 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.763992    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-secret-thanos-querier-tls\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.771141 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.771122    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9bj\" (UniqueName: \"kubernetes.io/projected/47253a1b-63ed-4ab6-9532-83eae4f7fdbd-kube-api-access-df9bj\") pod \"thanos-querier-6c676d7495-4qq7m\" (UID: \"47253a1b-63ed-4ab6-9532-83eae4f7fdbd\") " pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:26.794855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.794826    2565 generic.go:356] "Generic (PLEG): container finished" podID="aa52f873-c713-46b2-b054-0bce67a3111a" containerID="6474ca943262c905330dbca3ec0d8ec9c7f978963be63e594e93523bd6b92a5d" exitCode=0
Sep 07 21:03:26.794998 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.794907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxc5f" event={"ID":"aa52f873-c713-46b2-b054-0bce67a3111a","Type":"ContainerDied","Data":"6474ca943262c905330dbca3ec0d8ec9c7f978963be63e594e93523bd6b92a5d"}
Sep 07 21:03:26.796261 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.796242    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx" event={"ID":"8ef9a728-9e2b-4341-8554-7bcabaecdf7b","Type":"ContainerStarted","Data":"27b9febb567d424a43a00adc6bb17cd39fcb29c631a3aa0428e6ae60ef8e2ade"}
Sep 07 21:03:26.888151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:26.888109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:27.170359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.170290    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6c676d7495-4qq7m"]
Sep 07 21:03:27.240246 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:27.240210    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47253a1b_63ed_4ab6_9532_83eae4f7fdbd.slice/crio-ed59818ed09c17ea5ddcde3485a09f1e7400b0fca32a31c71af4fa7b2307b133 WatchSource:0}: Error finding container ed59818ed09c17ea5ddcde3485a09f1e7400b0fca32a31c71af4fa7b2307b133: Status 404 returned error can't find the container with id ed59818ed09c17ea5ddcde3485a09f1e7400b0fca32a31c71af4fa7b2307b133
Sep 07 21:03:27.802758 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.802680    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx" event={"ID":"8ef9a728-9e2b-4341-8554-7bcabaecdf7b","Type":"ContainerStarted","Data":"acb690459b01d03f18a8eada5aff2b3c42a28437295b4bc280cd4f7055db5483"}
Sep 07 21:03:27.802758 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.802742    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx" event={"ID":"8ef9a728-9e2b-4341-8554-7bcabaecdf7b","Type":"ContainerStarted","Data":"d9e2644e65746b47e054f845dbebdc28922d70bd10fae0d39d7b1052fdc80b78"}
Sep 07 21:03:27.803234 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.802770    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx" event={"ID":"8ef9a728-9e2b-4341-8554-7bcabaecdf7b","Type":"ContainerStarted","Data":"90475e9130188970e93120463b9c687ca36f71519132468c5ecab2f9d0226f65"}
Sep 07 21:03:27.804861 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.804839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxc5f" event={"ID":"aa52f873-c713-46b2-b054-0bce67a3111a","Type":"ContainerStarted","Data":"496f33295155cb378a507fc4501fcf62ed4c0f397d67b729f67d245f05502da9"}
Sep 07 21:03:27.804978 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.804882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sxc5f" event={"ID":"aa52f873-c713-46b2-b054-0bce67a3111a","Type":"ContainerStarted","Data":"fe9f09b7effab5aae77b94efc54245439ffd6abf92472b763e6040560ee9e01d"}
Sep 07 21:03:27.806003 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.805981    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m" event={"ID":"47253a1b-63ed-4ab6-9532-83eae4f7fdbd","Type":"ContainerStarted","Data":"ed59818ed09c17ea5ddcde3485a09f1e7400b0fca32a31c71af4fa7b2307b133"}
Sep 07 21:03:27.826063 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.826008    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-74cb57df7-nspqx" podStartSLOduration=2.386272036 podStartE2EDuration="3.8259892s" totalImagesPullingTime="1.439717164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:27.825318479 +0000 UTC m=+99.906845822" watchObservedRunningTime="2026-09-07 21:03:27.8259892 +0000 UTC m=+99.907516544"
Sep 07 21:03:27.846889 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:27.846838    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sxc5f" podStartSLOduration=2.722087947 podStartE2EDuration="3.846819648s" totalImagesPullingTime="1.064548373s" totalInitContainerRuntime="60.183328ms" isStatefulPod=true podCreationTimestamp="2026-09-07 21:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:27.846267645 +0000 UTC m=+99.927794989" watchObservedRunningTime="2026-09-07 21:03:27.846819648 +0000 UTC m=+99.928346989"
Sep 07 21:03:28.833909 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.833866    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-787485f48c-tnzkv"]
Sep 07 21:03:28.837423 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.837395    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:28.840114 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.840087    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 07 21:03:28.840114 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.840103    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 07 21:03:28.840947 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.840925    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 07 21:03:28.841054 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.840973    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-kust1lt8eqr1\""
Sep 07 21:03:28.841054 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.841009    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 07 21:03:28.841054 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.841040    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8nmvc\""
Sep 07 21:03:28.848341 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.848289    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-787485f48c-tnzkv"]
Sep 07 21:03:28.979490 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.979457    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-client-ca-bundle\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:28.979678 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.979522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-secret-metrics-server-tls\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:28.979678 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.979656    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-secret-metrics-server-client-certs\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:28.979864 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.979837    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/899d0185-7c8b-4477-991d-a369cc0fd262-audit-log\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:28.980013 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.979996    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/899d0185-7c8b-4477-991d-a369cc0fd262-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:28.980097 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.980032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/899d0185-7c8b-4477-991d-a369cc0fd262-metrics-server-audit-profiles\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:28.980160 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:28.980108    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpgc\" (UniqueName: \"kubernetes.io/projected/899d0185-7c8b-4477-991d-a369cc0fd262-kube-api-access-xzpgc\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.081554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.081493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-secret-metrics-server-client-certs\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.081554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.081549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/899d0185-7c8b-4477-991d-a369cc0fd262-audit-log\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.081789 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.081593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/899d0185-7c8b-4477-991d-a369cc0fd262-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.081789 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.081617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/899d0185-7c8b-4477-991d-a369cc0fd262-metrics-server-audit-profiles\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.081789 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.081672    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpgc\" (UniqueName: \"kubernetes.io/projected/899d0185-7c8b-4477-991d-a369cc0fd262-kube-api-access-xzpgc\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.081789 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.081712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-client-ca-bundle\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.081789 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.081747    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-secret-metrics-server-tls\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.082117 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.082090    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/899d0185-7c8b-4477-991d-a369cc0fd262-audit-log\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.082780 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.082527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/899d0185-7c8b-4477-991d-a369cc0fd262-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.082980 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.082938    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/899d0185-7c8b-4477-991d-a369cc0fd262-metrics-server-audit-profiles\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.084449 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.084391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-secret-metrics-server-tls\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.084547 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.084451    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-secret-metrics-server-client-certs\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.084590 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.084549    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/899d0185-7c8b-4477-991d-a369cc0fd262-client-ca-bundle\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.089765 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.089745    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpgc\" (UniqueName: \"kubernetes.io/projected/899d0185-7c8b-4477-991d-a369cc0fd262-kube-api-access-xzpgc\") pod \"metrics-server-787485f48c-tnzkv\" (UID: \"899d0185-7c8b-4477-991d-a369cc0fd262\") " pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.150407 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.150371    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:29.333843 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.333802    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-99c756565-sdmsf"]
Sep 07 21:03:29.336851 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.336799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.339394 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.339351    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-wrrkg\""
Sep 07 21:03:29.339481 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.339449    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 21:03:29.339760 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.339739    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 21:03:29.340299 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.340265    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 21:03:29.340393 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.340360    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 21:03:29.340498 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.340478    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 21:03:29.340562 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.340486    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 21:03:29.340636 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.340612    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 21:03:29.345922 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.345900    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 21:03:29.348811 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.348791    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99c756565-sdmsf"]
Sep 07 21:03:29.486119 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.486092    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-console-config\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.486220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.486127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-service-ca\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.486220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.486146    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-oauth-serving-cert\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.486220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.486170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-serving-cert\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.486220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.486212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-trusted-ca-bundle\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.486345 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.486230    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-oauth-config\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.486345 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.486250    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw6s9\" (UniqueName: \"kubernetes.io/projected/d794c107-bb76-40de-bdba-08cfae994012-kube-api-access-zw6s9\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.587445 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.587374    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-console-config\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.587445 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.587413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-service-ca\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.587627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.587575    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-oauth-serving-cert\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.587627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.587615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-serving-cert\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.587733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.587659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-trusted-ca-bundle\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.587733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.587680    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-oauth-config\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.587733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.587701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw6s9\" (UniqueName: \"kubernetes.io/projected/d794c107-bb76-40de-bdba-08cfae994012-kube-api-access-zw6s9\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.588463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.588222    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-service-ca\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.588463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.588361    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-oauth-serving-cert\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.588463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.588406    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-console-config\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.588887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.588839    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-trusted-ca-bundle\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.590263 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.590243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-oauth-config\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.590463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.590442    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-serving-cert\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.595507 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.595488    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw6s9\" (UniqueName: \"kubernetes.io/projected/d794c107-bb76-40de-bdba-08cfae994012-kube-api-access-zw6s9\") pod \"console-99c756565-sdmsf\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") " pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.602380 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.602358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-787485f48c-tnzkv"]
Sep 07 21:03:29.604690 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:29.604670    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899d0185_7c8b_4477_991d_a369cc0fd262.slice/crio-5dd6dd312b2bf5c75f0da4c950845b38e97a634355eccf4ebdd16fe9e0009a12 WatchSource:0}: Error finding container 5dd6dd312b2bf5c75f0da4c950845b38e97a634355eccf4ebdd16fe9e0009a12: Status 404 returned error can't find the container with id 5dd6dd312b2bf5c75f0da4c950845b38e97a634355eccf4ebdd16fe9e0009a12
Sep 07 21:03:29.654474 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.654448    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:29.816349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.816310    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m" event={"ID":"47253a1b-63ed-4ab6-9532-83eae4f7fdbd","Type":"ContainerStarted","Data":"cda78686e01b4964819735f39945c38b4067d6636c4d891b6464dcedc8633fa9"}
Sep 07 21:03:29.816515 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.816394    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m" event={"ID":"47253a1b-63ed-4ab6-9532-83eae4f7fdbd","Type":"ContainerStarted","Data":"22d14185e99034d57bed8664e8da4c415e74c180b1ab596df16b42dd67682c42"}
Sep 07 21:03:29.816515 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.816423    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m" event={"ID":"47253a1b-63ed-4ab6-9532-83eae4f7fdbd","Type":"ContainerStarted","Data":"be47a7a211ef0b21ef407e7544d21d1eac25696fd2a655741d8a3da136ebb004"}
Sep 07 21:03:29.817554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.817530    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv" event={"ID":"899d0185-7c8b-4477-991d-a369cc0fd262","Type":"ContainerStarted","Data":"5dd6dd312b2bf5c75f0da4c950845b38e97a634355eccf4ebdd16fe9e0009a12"}
Sep 07 21:03:29.924530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:29.924454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-99c756565-sdmsf"]
Sep 07 21:03:29.927059 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:29.927022    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd794c107_bb76_40de_bdba_08cfae994012.slice/crio-7cf2f77fa280499038048353d5c43fce773ae44769a23177a52069851811a148 WatchSource:0}: Error finding container 7cf2f77fa280499038048353d5c43fce773ae44769a23177a52069851811a148: Status 404 returned error can't find the container with id 7cf2f77fa280499038048353d5c43fce773ae44769a23177a52069851811a148
Sep 07 21:03:30.691282 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:30.690647    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ssk4q"
Sep 07 21:03:30.827993 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:30.827881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m" event={"ID":"47253a1b-63ed-4ab6-9532-83eae4f7fdbd","Type":"ContainerStarted","Data":"ed8c9fc264c979c5f36040a5ff9aae992c572e396e3a4a758cdc86996a923fbf"}
Sep 07 21:03:30.827993 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:30.827982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m" event={"ID":"47253a1b-63ed-4ab6-9532-83eae4f7fdbd","Type":"ContainerStarted","Data":"68a4da4e1b071a106b0abcec41e6927c38f6e3a3d8615c3b75446565193a06a1"}
Sep 07 21:03:30.829930 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:30.829905    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99c756565-sdmsf" event={"ID":"d794c107-bb76-40de-bdba-08cfae994012","Type":"ContainerStarted","Data":"7cf2f77fa280499038048353d5c43fce773ae44769a23177a52069851811a148"}
Sep 07 21:03:31.837431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:31.837392    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m" event={"ID":"47253a1b-63ed-4ab6-9532-83eae4f7fdbd","Type":"ContainerStarted","Data":"1a23ef2b6fb1e64bd956048e63a1ff1634efcc819699945eb7bfb3c9d6d9cb93"}
Sep 07 21:03:31.837862 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:31.837572    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:31.862745 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:31.862683    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m" podStartSLOduration=2.7007103900000002 podStartE2EDuration="5.862660591s" totalImagesPullingTime="3.161950201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:03:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:31.860083353 +0000 UTC m=+103.941610694" watchObservedRunningTime="2026-09-07 21:03:31.862660591 +0000 UTC m=+103.944187933"
Sep 07 21:03:33.848303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:33.848256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv" event={"ID":"899d0185-7c8b-4477-991d-a369cc0fd262","Type":"ContainerStarted","Data":"82c2a889a279685fa14b54f70720284e51d12ec891aece62fa5d4021eb814b21"}
Sep 07 21:03:33.849949 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:33.849925    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99c756565-sdmsf" event={"ID":"d794c107-bb76-40de-bdba-08cfae994012","Type":"ContainerStarted","Data":"dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570"}
Sep 07 21:03:33.867621 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:33.867566    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv" podStartSLOduration=3.82360872 podStartE2EDuration="5.86755147s" totalImagesPullingTime="2.04394275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:03:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:33.86571632 +0000 UTC m=+105.947243663" watchObservedRunningTime="2026-09-07 21:03:33.86755147 +0000 UTC m=+105.949078855"
Sep 07 21:03:33.886588 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:33.886538    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-99c756565-sdmsf" podStartSLOduration=1.722883221 podStartE2EDuration="4.886521904s" totalImagesPullingTime="3.163638683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:03:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:03:33.885633561 +0000 UTC m=+105.967160905" watchObservedRunningTime="2026-09-07 21:03:33.886521904 +0000 UTC m=+105.968049245"
Sep 07 21:03:37.848860 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:37.848826    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6c676d7495-4qq7m"
Sep 07 21:03:39.655354 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:39.655325    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:39.655735 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:39.655370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:39.660086 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:39.660059    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:39.876812 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:39.876784    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:03:49.150528 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:49.150496    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:49.151052 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:49.150588    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:03:58.124703 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.124665    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:03:58.126910 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.126890    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8-metrics-certs\") pod \"network-metrics-daemon-xbp8h\" (UID: \"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8\") " pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:03:58.185204 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.185172    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9r8wd\""
Sep 07 21:03:58.192909 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.192887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xbp8h"
Sep 07 21:03:58.465893 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.465863    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xbp8h"]
Sep 07 21:03:58.468489 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:03:58.468464    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2499acf3_9ad9_4ddd_9c3d_c7bedecd1ac8.slice/crio-03b9788a5dda17ed27df1069aec00a8b3209f0ef8fc55e1abcc277a0bf3680f5 WatchSource:0}: Error finding container 03b9788a5dda17ed27df1069aec00a8b3209f0ef8fc55e1abcc277a0bf3680f5: Status 404 returned error can't find the container with id 03b9788a5dda17ed27df1069aec00a8b3209f0ef8fc55e1abcc277a0bf3680f5
Sep 07 21:03:58.942342 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.942312    2565 generic.go:356] "Generic (PLEG): container finished" podID="c9b1579c-00e4-4922-8bd5-87f58f77c496" containerID="adc67c784d2ae1c4c3962ee9f92463b61d18e4035b48102b33248baf9ab3a988" exitCode=0
Sep 07 21:03:58.942509 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.942395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv" event={"ID":"c9b1579c-00e4-4922-8bd5-87f58f77c496","Type":"ContainerDied","Data":"adc67c784d2ae1c4c3962ee9f92463b61d18e4035b48102b33248baf9ab3a988"}
Sep 07 21:03:58.942876 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.942855    2565 scope.go:117] "RemoveContainer" containerID="adc67c784d2ae1c4c3962ee9f92463b61d18e4035b48102b33248baf9ab3a988"
Sep 07 21:03:58.943906 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:58.943850    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbp8h" event={"ID":"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8","Type":"ContainerStarted","Data":"03b9788a5dda17ed27df1069aec00a8b3209f0ef8fc55e1abcc277a0bf3680f5"}
Sep 07 21:03:59.949057 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:59.949019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-qb7kv" event={"ID":"c9b1579c-00e4-4922-8bd5-87f58f77c496","Type":"ContainerStarted","Data":"7586b33dd17ca0c2759d187feabd1213f3b1f433b70292666d9ae3f3d38ef87f"}
Sep 07 21:03:59.950739 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:59.950715    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbp8h" event={"ID":"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8","Type":"ContainerStarted","Data":"95ee2ae33fd43c7860205e726bf054c4d08214be389114005dce1b7412ee9c01"}
Sep 07 21:03:59.950863 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:03:59.950756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xbp8h" event={"ID":"2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8","Type":"ContainerStarted","Data":"11a48b8b0be8782874ca967ed47acf5a65f1568aa6e379a92124530201e981c4"}
Sep 07 21:04:00.009152 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:00.009099    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xbp8h" podStartSLOduration=130.913857265 podStartE2EDuration="2m12.009084148s" totalImagesPullingTime="1.095226883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:01:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:04:00.007743587 +0000 UTC m=+132.089270940" watchObservedRunningTime="2026-09-07 21:04:00.009084148 +0000 UTC m=+132.090611489"
Sep 07 21:04:09.156192 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:09.156157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:04:09.159942 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:09.159921    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-787485f48c-tnzkv"
Sep 07 21:04:51.836490 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.836452    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67f7bd7db6-mmfhk"]
Sep 07 21:04:51.840302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.840279    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:51.851207 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.851182    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f7bd7db6-mmfhk"]
Sep 07 21:04:51.922219 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.922187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47cck\" (UniqueName: \"kubernetes.io/projected/42c1966a-7e35-47ee-b258-d74ed4aae474-kube-api-access-47cck\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:51.922374 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.922238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-oauth-config\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:51.922374 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.922324    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-oauth-serving-cert\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:51.922449 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.922382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-serving-cert\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:51.922449 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.922403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-service-ca\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:51.922449 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.922421    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-console-config\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:51.922449 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:51.922436    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-trusted-ca-bundle\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.023136 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.023093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47cck\" (UniqueName: \"kubernetes.io/projected/42c1966a-7e35-47ee-b258-d74ed4aae474-kube-api-access-47cck\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.023136 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.023145    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-oauth-config\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.023306 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.023180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-oauth-serving-cert\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.023306 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.023218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-serving-cert\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.023391 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.023365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-service-ca\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.023450 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.023411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-console-config\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.023450 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.023433    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-trusted-ca-bundle\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.024044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.024018    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-oauth-serving-cert\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.024220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.024080    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-service-ca\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.024311 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.024162    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-console-config\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.024311 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.024302    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-trusted-ca-bundle\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.025583 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.025563    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-serving-cert\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.025736 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.025717    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-oauth-config\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.030493 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.030475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47cck\" (UniqueName: \"kubernetes.io/projected/42c1966a-7e35-47ee-b258-d74ed4aae474-kube-api-access-47cck\") pod \"console-67f7bd7db6-mmfhk\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") " pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.151148 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.151072    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:04:52.422313 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:52.422238    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67f7bd7db6-mmfhk"]
Sep 07 21:04:52.424922 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:04:52.424894    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c1966a_7e35_47ee_b258_d74ed4aae474.slice/crio-72b1edd2868e5cfe83888a0fc016d20c1ac6df5c0e959913e72d260945c1d5ca WatchSource:0}: Error finding container 72b1edd2868e5cfe83888a0fc016d20c1ac6df5c0e959913e72d260945c1d5ca: Status 404 returned error can't find the container with id 72b1edd2868e5cfe83888a0fc016d20c1ac6df5c0e959913e72d260945c1d5ca
Sep 07 21:04:53.155200 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:53.155167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7bd7db6-mmfhk" event={"ID":"42c1966a-7e35-47ee-b258-d74ed4aae474","Type":"ContainerStarted","Data":"679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4"}
Sep 07 21:04:53.155597 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:53.155222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7bd7db6-mmfhk" event={"ID":"42c1966a-7e35-47ee-b258-d74ed4aae474","Type":"ContainerStarted","Data":"72b1edd2868e5cfe83888a0fc016d20c1ac6df5c0e959913e72d260945c1d5ca"}
Sep 07 21:04:53.173063 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:04:53.173014    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67f7bd7db6-mmfhk" podStartSLOduration=2.173000577 podStartE2EDuration="2.173000577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:04:53.170836515 +0000 UTC m=+185.252363856" watchObservedRunningTime="2026-09-07 21:04:53.173000577 +0000 UTC m=+185.254527918"
Sep 07 21:05:02.151283 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:02.151239    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:05:02.151719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:02.151316    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:05:02.156094 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:02.156068    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:05:02.193017 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:02.192988    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:05:02.236719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:02.236677    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-99c756565-sdmsf"]
Sep 07 21:05:27.258628 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.258584    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-99c756565-sdmsf" podUID="d794c107-bb76-40de-bdba-08cfae994012" containerName="console" containerID="cri-o://dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570" gracePeriod=15
Sep 07 21:05:27.494350 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.494327    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:05:27.592040 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592015    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-console-config\") pod \"d794c107-bb76-40de-bdba-08cfae994012\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") "
Sep 07 21:05:27.592183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592074    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zw6s9\" (UniqueName: \"kubernetes.io/projected/d794c107-bb76-40de-bdba-08cfae994012-kube-api-access-zw6s9\") pod \"d794c107-bb76-40de-bdba-08cfae994012\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") "
Sep 07 21:05:27.592183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592101    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-oauth-config\") pod \"d794c107-bb76-40de-bdba-08cfae994012\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") "
Sep 07 21:05:27.592183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592121    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-oauth-serving-cert\") pod \"d794c107-bb76-40de-bdba-08cfae994012\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") "
Sep 07 21:05:27.592183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592142    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-service-ca\") pod \"d794c107-bb76-40de-bdba-08cfae994012\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") "
Sep 07 21:05:27.592183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-trusted-ca-bundle\") pod \"d794c107-bb76-40de-bdba-08cfae994012\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") "
Sep 07 21:05:27.592483 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-serving-cert\") pod \"d794c107-bb76-40de-bdba-08cfae994012\" (UID: \"d794c107-bb76-40de-bdba-08cfae994012\") "
Sep 07 21:05:27.592543 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592479    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-console-config" (OuterVolumeSpecName: "console-config") pod "d794c107-bb76-40de-bdba-08cfae994012" (UID: "d794c107-bb76-40de-bdba-08cfae994012"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:05:27.592613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592585    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d794c107-bb76-40de-bdba-08cfae994012" (UID: "d794c107-bb76-40de-bdba-08cfae994012"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:05:27.592663 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592635    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d794c107-bb76-40de-bdba-08cfae994012" (UID: "d794c107-bb76-40de-bdba-08cfae994012"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:05:27.592718 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.592679    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-service-ca" (OuterVolumeSpecName: "service-ca") pod "d794c107-bb76-40de-bdba-08cfae994012" (UID: "d794c107-bb76-40de-bdba-08cfae994012"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:05:27.594517 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.594492    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d794c107-bb76-40de-bdba-08cfae994012" (UID: "d794c107-bb76-40de-bdba-08cfae994012"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:05:27.594621 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.594512    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d794c107-bb76-40de-bdba-08cfae994012" (UID: "d794c107-bb76-40de-bdba-08cfae994012"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:05:27.594621 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.594497    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d794c107-bb76-40de-bdba-08cfae994012-kube-api-access-zw6s9" (OuterVolumeSpecName: "kube-api-access-zw6s9") pod "d794c107-bb76-40de-bdba-08cfae994012" (UID: "d794c107-bb76-40de-bdba-08cfae994012"). InnerVolumeSpecName "kube-api-access-zw6s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:05:27.692870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.692844    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-serving-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:05:27.692870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.692866    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-console-config\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:05:27.692870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.692875    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw6s9\" (UniqueName: \"kubernetes.io/projected/d794c107-bb76-40de-bdba-08cfae994012-kube-api-access-zw6s9\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:05:27.693075 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.692884    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d794c107-bb76-40de-bdba-08cfae994012-console-oauth-config\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:05:27.693075 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.692892    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-oauth-serving-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:05:27.693075 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.692902    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:05:27.693075 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:27.692910    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d794c107-bb76-40de-bdba-08cfae994012-trusted-ca-bundle\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:05:28.284928 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.284892    2565 generic.go:356] "Generic (PLEG): container finished" podID="d794c107-bb76-40de-bdba-08cfae994012" containerID="dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570" exitCode=0
Sep 07 21:05:28.284928 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.284931    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-99c756565-sdmsf"
Sep 07 21:05:28.285439 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.284984    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99c756565-sdmsf" event={"ID":"d794c107-bb76-40de-bdba-08cfae994012","Type":"ContainerDied","Data":"dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570"}
Sep 07 21:05:28.285439 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.285034    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-99c756565-sdmsf" event={"ID":"d794c107-bb76-40de-bdba-08cfae994012","Type":"ContainerDied","Data":"7cf2f77fa280499038048353d5c43fce773ae44769a23177a52069851811a148"}
Sep 07 21:05:28.285439 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.285055    2565 scope.go:117] "RemoveContainer" containerID="dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570"
Sep 07 21:05:28.293504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.293489    2565 scope.go:117] "RemoveContainer" containerID="dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570"
Sep 07 21:05:28.293784 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:05:28.293764    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570\": container with ID starting with dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570 not found: ID does not exist" containerID="dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570"
Sep 07 21:05:28.293839 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.293791    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570"} err="failed to get container status \"dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570\": rpc error: code = NotFound desc = could not find container \"dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570\": container with ID starting with dc9482eba4adeec629f99ecfc614dbb6470cdc4adf4eecf4498c5de94a784570 not found: ID does not exist"
Sep 07 21:05:28.304554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.304534    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-99c756565-sdmsf"]
Sep 07 21:05:28.308318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.308298    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-99c756565-sdmsf"]
Sep 07 21:05:28.363183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:28.363149    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d794c107-bb76-40de-bdba-08cfae994012" path="/var/lib/kubelet/pods/d794c107-bb76-40de-bdba-08cfae994012/volumes"
Sep 07 21:05:59.698743 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.698705    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cf56d8966-4mjsf"]
Sep 07 21:05:59.699416 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.699027    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d794c107-bb76-40de-bdba-08cfae994012" containerName="console"
Sep 07 21:05:59.699416 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.699040    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d794c107-bb76-40de-bdba-08cfae994012" containerName="console"
Sep 07 21:05:59.699416 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.699095    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d794c107-bb76-40de-bdba-08cfae994012" containerName="console"
Sep 07 21:05:59.703351 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.703324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.712628 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.712606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf56d8966-4mjsf"]
Sep 07 21:05:59.856888 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.856846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-config\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.856888 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.856888    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-service-ca\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.857146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.856920    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6779f\" (UniqueName: \"kubernetes.io/projected/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-kube-api-access-6779f\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.857146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.857041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-trusted-ca-bundle\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.857146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.857077    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-oauth-serving-cert\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.857146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.857095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-oauth-config\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.857304 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.857173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-serving-cert\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.957647 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.957547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-serving-cert\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.957647 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.957609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-config\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.957647 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.957628    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-service-ca\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.957647 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.957655    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6779f\" (UniqueName: \"kubernetes.io/projected/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-kube-api-access-6779f\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.958005 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.957681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-trusted-ca-bundle\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.958005 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.957707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-oauth-serving-cert\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.958005 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.957733    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-oauth-config\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.958555 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.958524    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-config\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.958669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.958532    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-service-ca\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.958669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.958569    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-oauth-serving-cert\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.958669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.958639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-trusted-ca-bundle\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.960089 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.960065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-serving-cert\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.960212 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.960195    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-oauth-config\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:05:59.964919 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:05:59.964901    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6779f\" (UniqueName: \"kubernetes.io/projected/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-kube-api-access-6779f\") pod \"console-6cf56d8966-4mjsf\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") " pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:06:00.011937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:00.011901    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:06:00.280768 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:00.280685    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cf56d8966-4mjsf"]
Sep 07 21:06:00.283565 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:06:00.283536    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fcbd0a_ad4e_4d24_862a_08fca9ab69fb.slice/crio-a65b82544ec434561cd325ed7b663bd0d47966a0cc4ea226d4d9e785911a5c96 WatchSource:0}: Error finding container a65b82544ec434561cd325ed7b663bd0d47966a0cc4ea226d4d9e785911a5c96: Status 404 returned error can't find the container with id a65b82544ec434561cd325ed7b663bd0d47966a0cc4ea226d4d9e785911a5c96
Sep 07 21:06:00.410578 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:00.410544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf56d8966-4mjsf" event={"ID":"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb","Type":"ContainerStarted","Data":"82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba"}
Sep 07 21:06:00.410758 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:00.410599    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf56d8966-4mjsf" event={"ID":"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb","Type":"ContainerStarted","Data":"a65b82544ec434561cd325ed7b663bd0d47966a0cc4ea226d4d9e785911a5c96"}
Sep 07 21:06:00.427426 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:00.427374    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cf56d8966-4mjsf" podStartSLOduration=1.427357964 podStartE2EDuration="1.427357964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:05:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:06:00.426592208 +0000 UTC m=+252.508119550" watchObservedRunningTime="2026-09-07 21:06:00.427357964 +0000 UTC m=+252.508885305"
Sep 07 21:06:10.012954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:10.012894    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:06:10.012954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:10.012991    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:06:10.017887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:10.017860    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:06:10.452791 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:10.452762    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:06:10.496454 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:10.496422    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f7bd7db6-mmfhk"]
Sep 07 21:06:35.517501 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.517431    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-67f7bd7db6-mmfhk" podUID="42c1966a-7e35-47ee-b258-d74ed4aae474" containerName="console" containerID="cri-o://679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4" gracePeriod=15
Sep 07 21:06:35.754724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.754698    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:06:35.832177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832104    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-oauth-config\") pod \"42c1966a-7e35-47ee-b258-d74ed4aae474\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") "
Sep 07 21:06:35.832177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-service-ca\") pod \"42c1966a-7e35-47ee-b258-d74ed4aae474\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") "
Sep 07 21:06:35.832177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-serving-cert\") pod \"42c1966a-7e35-47ee-b258-d74ed4aae474\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") "
Sep 07 21:06:35.832431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832226    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47cck\" (UniqueName: \"kubernetes.io/projected/42c1966a-7e35-47ee-b258-d74ed4aae474-kube-api-access-47cck\") pod \"42c1966a-7e35-47ee-b258-d74ed4aae474\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") "
Sep 07 21:06:35.832431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832244    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-oauth-serving-cert\") pod \"42c1966a-7e35-47ee-b258-d74ed4aae474\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") "
Sep 07 21:06:35.832431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832276    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-console-config\") pod \"42c1966a-7e35-47ee-b258-d74ed4aae474\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") "
Sep 07 21:06:35.832431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-trusted-ca-bundle\") pod \"42c1966a-7e35-47ee-b258-d74ed4aae474\" (UID: \"42c1966a-7e35-47ee-b258-d74ed4aae474\") "
Sep 07 21:06:35.832634 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832557    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-service-ca" (OuterVolumeSpecName: "service-ca") pod "42c1966a-7e35-47ee-b258-d74ed4aae474" (UID: "42c1966a-7e35-47ee-b258-d74ed4aae474"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:06:35.832714 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832682    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-console-config" (OuterVolumeSpecName: "console-config") pod "42c1966a-7e35-47ee-b258-d74ed4aae474" (UID: "42c1966a-7e35-47ee-b258-d74ed4aae474"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:06:35.832841 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832733    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "42c1966a-7e35-47ee-b258-d74ed4aae474" (UID: "42c1966a-7e35-47ee-b258-d74ed4aae474"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:06:35.832841 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.832826    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "42c1966a-7e35-47ee-b258-d74ed4aae474" (UID: "42c1966a-7e35-47ee-b258-d74ed4aae474"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:06:35.834213 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.834192    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "42c1966a-7e35-47ee-b258-d74ed4aae474" (UID: "42c1966a-7e35-47ee-b258-d74ed4aae474"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:06:35.834340 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.834327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "42c1966a-7e35-47ee-b258-d74ed4aae474" (UID: "42c1966a-7e35-47ee-b258-d74ed4aae474"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:06:35.834440 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.834417    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42c1966a-7e35-47ee-b258-d74ed4aae474-kube-api-access-47cck" (OuterVolumeSpecName: "kube-api-access-47cck") pod "42c1966a-7e35-47ee-b258-d74ed4aae474" (UID: "42c1966a-7e35-47ee-b258-d74ed4aae474"). InnerVolumeSpecName "kube-api-access-47cck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:06:35.932839 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.932811    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47cck\" (UniqueName: \"kubernetes.io/projected/42c1966a-7e35-47ee-b258-d74ed4aae474-kube-api-access-47cck\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:06:35.932839 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.932835    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-oauth-serving-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:06:35.932839 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.932845    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-console-config\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:06:35.933066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.932854    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-trusted-ca-bundle\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:06:35.933066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.932862    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-oauth-config\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:06:35.933066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.932872    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/42c1966a-7e35-47ee-b258-d74ed4aae474-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:06:35.933066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:35.932880    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/42c1966a-7e35-47ee-b258-d74ed4aae474-console-serving-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:06:36.546011 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.545974    2565 generic.go:356] "Generic (PLEG): container finished" podID="42c1966a-7e35-47ee-b258-d74ed4aae474" containerID="679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4" exitCode=0
Sep 07 21:06:36.546011 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.546013    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67f7bd7db6-mmfhk"
Sep 07 21:06:36.546530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.546057    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7bd7db6-mmfhk" event={"ID":"42c1966a-7e35-47ee-b258-d74ed4aae474","Type":"ContainerDied","Data":"679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4"}
Sep 07 21:06:36.546530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.546105    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67f7bd7db6-mmfhk" event={"ID":"42c1966a-7e35-47ee-b258-d74ed4aae474","Type":"ContainerDied","Data":"72b1edd2868e5cfe83888a0fc016d20c1ac6df5c0e959913e72d260945c1d5ca"}
Sep 07 21:06:36.546530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.546127    2565 scope.go:117] "RemoveContainer" containerID="679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4"
Sep 07 21:06:36.554085 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.554065    2565 scope.go:117] "RemoveContainer" containerID="679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4"
Sep 07 21:06:36.554341 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:06:36.554325    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4\": container with ID starting with 679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4 not found: ID does not exist" containerID="679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4"
Sep 07 21:06:36.554391 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.554347    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4"} err="failed to get container status \"679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4\": rpc error: code = NotFound desc = could not find container \"679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4\": container with ID starting with 679e64ecb79d52410a8fc544b4e0c5be8acadce6d8a42a0a7b6a9eeb808075b4 not found: ID does not exist"
Sep 07 21:06:36.561103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.561081    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-67f7bd7db6-mmfhk"]
Sep 07 21:06:36.564710 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:36.564692    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-67f7bd7db6-mmfhk"]
Sep 07 21:06:38.362486 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:38.362455    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42c1966a-7e35-47ee-b258-d74ed4aae474" path="/var/lib/kubelet/pods/42c1966a-7e35-47ee-b258-d74ed4aae474/volumes"
Sep 07 21:06:48.328225 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:06:48.328196    2565 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 21:07:41.471075 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.470992    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"]
Sep 07 21:07:41.471457 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.471275    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42c1966a-7e35-47ee-b258-d74ed4aae474" containerName="console"
Sep 07 21:07:41.471457 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.471286    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42c1966a-7e35-47ee-b258-d74ed4aae474" containerName="console"
Sep 07 21:07:41.471457 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.471332    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42c1966a-7e35-47ee-b258-d74ed4aae474" containerName="console"
Sep 07 21:07:41.474322 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.474304    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.477122 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.477101    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wgpwk\""
Sep 07 21:07:41.478082 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.478059    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 21:07:41.478211 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.478192    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 21:07:41.478271 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.478191    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 21:07:41.478271 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.478238    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 21:07:41.492742 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.492720    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"]
Sep 07 21:07:41.546380 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.546354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e870ee56-e294-4550-9a9b-352b31377554-manifests\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.546522 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.546406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e870ee56-e294-4550-9a9b-352b31377554-webhook-cert\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.546522 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.546506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e870ee56-e294-4550-9a9b-352b31377554-tmp\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.546630 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.546537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e870ee56-e294-4550-9a9b-352b31377554-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.546630 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.546554    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjv9g\" (UniqueName: \"kubernetes.io/projected/e870ee56-e294-4550-9a9b-352b31377554-kube-api-access-jjv9g\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.647254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.647224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e870ee56-e294-4550-9a9b-352b31377554-tmp\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.647254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.647257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e870ee56-e294-4550-9a9b-352b31377554-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.647460 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.647273    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjv9g\" (UniqueName: \"kubernetes.io/projected/e870ee56-e294-4550-9a9b-352b31377554-kube-api-access-jjv9g\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.647460 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.647308    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e870ee56-e294-4550-9a9b-352b31377554-manifests\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.647460 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.647350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e870ee56-e294-4550-9a9b-352b31377554-webhook-cert\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.647700 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.647681    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e870ee56-e294-4550-9a9b-352b31377554-manifests\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.649549 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.649521    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e870ee56-e294-4550-9a9b-352b31377554-tmp\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.649745 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.649727    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e870ee56-e294-4550-9a9b-352b31377554-apiservice-cert\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.649882 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.649860    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e870ee56-e294-4550-9a9b-352b31377554-webhook-cert\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.655047 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.655030    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjv9g\" (UniqueName: \"kubernetes.io/projected/e870ee56-e294-4550-9a9b-352b31377554-kube-api-access-jjv9g\") pod \"opendatahub-operator-controller-manager-56489b9975-275kj\" (UID: \"e870ee56-e294-4550-9a9b-352b31377554\") " pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:41.783663 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:41.783574    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:42.057491 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:42.057296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"]
Sep 07 21:07:42.060473 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:07:42.060445    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode870ee56_e294_4550_9a9b_352b31377554.slice/crio-8afe40cd1930057e8052a9a6f22b32745a5817a8bf5f8662f21237dafff8e714 WatchSource:0}: Error finding container 8afe40cd1930057e8052a9a6f22b32745a5817a8bf5f8662f21237dafff8e714: Status 404 returned error can't find the container with id 8afe40cd1930057e8052a9a6f22b32745a5817a8bf5f8662f21237dafff8e714
Sep 07 21:07:42.062186 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:42.062166    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 21:07:42.789633 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:42.789590    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj" event={"ID":"e870ee56-e294-4550-9a9b-352b31377554","Type":"ContainerStarted","Data":"8afe40cd1930057e8052a9a6f22b32745a5817a8bf5f8662f21237dafff8e714"}
Sep 07 21:07:45.805880 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:45.805837    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj" event={"ID":"e870ee56-e294-4550-9a9b-352b31377554","Type":"ContainerStarted","Data":"83fbd8d53444b647e6f3105c7e5b06a1ff173cae157a98defaafb631008ecbfa"}
Sep 07 21:07:46.810762 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:46.810729    2565 generic.go:356] "Generic (PLEG): container finished" podID="e870ee56-e294-4550-9a9b-352b31377554" containerID="83fbd8d53444b647e6f3105c7e5b06a1ff173cae157a98defaafb631008ecbfa" exitCode=0
Sep 07 21:07:46.810762 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:46.810767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj" event={"ID":"e870ee56-e294-4550-9a9b-352b31377554","Type":"ContainerDied","Data":"83fbd8d53444b647e6f3105c7e5b06a1ff173cae157a98defaafb631008ecbfa"}
Sep 07 21:07:47.816377 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:47.816340    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj" event={"ID":"e870ee56-e294-4550-9a9b-352b31377554","Type":"ContainerStarted","Data":"2ad49b4afd03b9829d7efd3912ab0ca9b2e2a22396e63c72c1bb7e3f0f730da3"}
Sep 07 21:07:47.816754 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:47.816444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:07:47.839872 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:47.839825    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj" podStartSLOduration=2.341503052 podStartE2EDuration="6.839809556s" totalImagesPullingTime="3.411663945s" totalInitContainerRuntime="1.086642559s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:07:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:07:47.838007036 +0000 UTC m=+359.919534376" watchObservedRunningTime="2026-09-07 21:07:47.839809556 +0000 UTC m=+359.921336897"
Sep 07 21:07:58.821956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:07:58.821926    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-56489b9975-275kj"
Sep 07 21:08:00.849607 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:00.849571    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"]
Sep 07 21:08:00.863291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:00.863263    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:00.866253 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:00.866228    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 07 21:08:00.866397 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:00.866270    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-t6cx2\""
Sep 07 21:08:00.866505 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:00.866491    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 07 21:08:00.872058 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:00.872035    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"]
Sep 07 21:08:01.006557 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.006519    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146296e4-520f-4615-b9f8-3127417fb4de-tls-certs\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.006725 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.006588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146296e4-520f-4615-b9f8-3127417fb4de-tmp\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.006725 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.006674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpnb2\" (UniqueName: \"kubernetes.io/projected/146296e4-520f-4615-b9f8-3127417fb4de-kube-api-access-wpnb2\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.107448 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.107348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146296e4-520f-4615-b9f8-3127417fb4de-tmp\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.107448 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.107422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpnb2\" (UniqueName: \"kubernetes.io/projected/146296e4-520f-4615-b9f8-3127417fb4de-kube-api-access-wpnb2\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.107647 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.107467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146296e4-520f-4615-b9f8-3127417fb4de-tls-certs\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.107647 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:08:01.107597    2565 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 07 21:08:01.107725 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:08:01.107674    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/146296e4-520f-4615-b9f8-3127417fb4de-tls-certs podName:146296e4-520f-4615-b9f8-3127417fb4de nodeName:}" failed. No retries permitted until 2026-09-07 21:08:01.607657855 +0000 UTC m=+373.689185178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/146296e4-520f-4615-b9f8-3127417fb4de-tls-certs") pod "kube-auth-proxy-58b8d644dc-tc66s" (UID: "146296e4-520f-4615-b9f8-3127417fb4de") : secret "kube-auth-proxy-tls" not found
Sep 07 21:08:01.109546 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.109528    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/146296e4-520f-4615-b9f8-3127417fb4de-tmp\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.115832 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.115813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpnb2\" (UniqueName: \"kubernetes.io/projected/146296e4-520f-4615-b9f8-3127417fb4de-kube-api-access-wpnb2\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.611885 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.611848    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146296e4-520f-4615-b9f8-3127417fb4de-tls-certs\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.614361 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.614338    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/146296e4-520f-4615-b9f8-3127417fb4de-tls-certs\") pod \"kube-auth-proxy-58b8d644dc-tc66s\" (UID: \"146296e4-520f-4615-b9f8-3127417fb4de\") " pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.772940 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.772899    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"
Sep 07 21:08:01.945905 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.945831    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"]
Sep 07 21:08:01.950206 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.950187    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:01.952883 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.952861    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 07 21:08:01.952994 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.952895    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 07 21:08:01.952994 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.952933    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 07 21:08:01.953146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.953133    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 21:08:01.953493 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.953473    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-65gdm\""
Sep 07 21:08:01.962102 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:01.962077    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"]
Sep 07 21:08:02.042600 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.042576    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s"]
Sep 07 21:08:02.044627 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:08:02.044602    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146296e4_520f_4615_b9f8_3127417fb4de.slice/crio-58a28e737fbf72a586abef98aa154215604f99f67c5d38ebec57a9ec23979142 WatchSource:0}: Error finding container 58a28e737fbf72a586abef98aa154215604f99f67c5d38ebec57a9ec23979142: Status 404 returned error can't find the container with id 58a28e737fbf72a586abef98aa154215604f99f67c5d38ebec57a9ec23979142
Sep 07 21:08:02.116369 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.116344    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.116489 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.116372    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.116489 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.116392    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4fxs\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-kube-api-access-h4fxs\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.116579 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.116480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/139aa155-b33e-4c81-8b1e-546d5dffdd65-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.116579 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.116511    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.116579 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.116546    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.116579 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.116573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.217715 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.217617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.217715 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.217659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.217715 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.217688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4fxs\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-kube-api-access-h4fxs\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.218021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.217730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/139aa155-b33e-4c81-8b1e-546d5dffdd65-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.218021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.217758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.218021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.217802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.218021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.217835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.218362 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.218328    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.220085 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.220062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/139aa155-b33e-4c81-8b1e-546d5dffdd65-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.220206 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.220188    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.220549 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.220527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.220618 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.220540    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.225298 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.225267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.225754 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.225732    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4fxs\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-kube-api-access-h4fxs\") pod \"istiod-openshift-gateway-7b7995d5cd-gfkr2\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.259670 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.259643    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:02.531579 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.531452    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"]
Sep 07 21:08:02.534055 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:08:02.534015    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod139aa155_b33e_4c81_8b1e_546d5dffdd65.slice/crio-8570ff7ca1ce4d849d331772fc4fa58f1ff1628babe44de8cf3db9bfbdf87ec9 WatchSource:0}: Error finding container 8570ff7ca1ce4d849d331772fc4fa58f1ff1628babe44de8cf3db9bfbdf87ec9: Status 404 returned error can't find the container with id 8570ff7ca1ce4d849d331772fc4fa58f1ff1628babe44de8cf3db9bfbdf87ec9
Sep 07 21:08:02.876795 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.876748    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s" event={"ID":"146296e4-520f-4615-b9f8-3127417fb4de","Type":"ContainerStarted","Data":"58a28e737fbf72a586abef98aa154215604f99f67c5d38ebec57a9ec23979142"}
Sep 07 21:08:02.878706 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:02.878677    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2" event={"ID":"139aa155-b33e-4c81-8b1e-546d5dffdd65","Type":"ContainerStarted","Data":"8570ff7ca1ce4d849d331772fc4fa58f1ff1628babe44de8cf3db9bfbdf87ec9"}
Sep 07 21:08:05.894555 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:05.894517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s" event={"ID":"146296e4-520f-4615-b9f8-3127417fb4de","Type":"ContainerStarted","Data":"7a45bdcef88df9c7d6e1767eedb1e9bf1f8e66782b80e18adba6f7f472a3ebba"}
Sep 07 21:08:05.911271 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:05.911217    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58b8d644dc-tc66s" podStartSLOduration=3.069133453 podStartE2EDuration="5.911198002s" totalImagesPullingTime="2.842064549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:08:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:08:05.909680336 +0000 UTC m=+377.991207679" watchObservedRunningTime="2026-09-07 21:08:05.911198002 +0000 UTC m=+377.992725342"
Sep 07 21:08:06.708778 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:06.708743    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:08:06.708855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:06.708815    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:08:06.900076 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:06.900037    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2" event={"ID":"139aa155-b33e-4c81-8b1e-546d5dffdd65","Type":"ContainerStarted","Data":"4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636"}
Sep 07 21:08:06.920430 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:06.920369    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2" podStartSLOduration=1.7477862530000001 podStartE2EDuration="5.920354697s" totalImagesPullingTime="4.172568444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:08:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:08:06.918246255 +0000 UTC m=+378.999773596" watchObservedRunningTime="2026-09-07 21:08:06.920354697 +0000 UTC m=+379.001882037"
Sep 07 21:08:07.904801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:07.904773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:07.906638 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:07.906596    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-gfkr2 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 21:08:07.906771 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:07.906677    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2" podUID="139aa155-b33e-4c81-8b1e-546d5dffdd65" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 21:08:08.909244 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:08.909214    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:19.593477 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.593443    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"]
Sep 07 21:08:19.596706 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.596689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.600417 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.600391    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j5q6n\""
Sep 07 21:08:19.600582 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.600423    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 07 21:08:19.600706 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.600423    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 21:08:19.600768 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.600429    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 07 21:08:19.600768 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.600433    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 21:08:19.600864 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.600439    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 07 21:08:19.605908 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.605883    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"]
Sep 07 21:08:19.664488 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.664454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/75ebc97d-4abc-4a7e-bbea-a509781a7975-metrics-cert\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.664654 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.664529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/75ebc97d-4abc-4a7e-bbea-a509781a7975-manager-config\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.664654 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.664570    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnbvq\" (UniqueName: \"kubernetes.io/projected/75ebc97d-4abc-4a7e-bbea-a509781a7975-kube-api-access-jnbvq\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.664654 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.664646    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75ebc97d-4abc-4a7e-bbea-a509781a7975-cert\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.765579 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.765541    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75ebc97d-4abc-4a7e-bbea-a509781a7975-cert\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.765760 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.765595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/75ebc97d-4abc-4a7e-bbea-a509781a7975-metrics-cert\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.765760 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.765632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/75ebc97d-4abc-4a7e-bbea-a509781a7975-manager-config\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.765760 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.765676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbvq\" (UniqueName: \"kubernetes.io/projected/75ebc97d-4abc-4a7e-bbea-a509781a7975-kube-api-access-jnbvq\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.766350 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.766323    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/75ebc97d-4abc-4a7e-bbea-a509781a7975-manager-config\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.768162 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.768138    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/75ebc97d-4abc-4a7e-bbea-a509781a7975-metrics-cert\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.768242 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.768183    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/75ebc97d-4abc-4a7e-bbea-a509781a7975-cert\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.774845 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.774824    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnbvq\" (UniqueName: \"kubernetes.io/projected/75ebc97d-4abc-4a7e-bbea-a509781a7975-kube-api-access-jnbvq\") pod \"lws-controller-manager-75546db857-h8g6p\" (UID: \"75ebc97d-4abc-4a7e-bbea-a509781a7975\") " pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:19.906954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:19.906871    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:20.191352 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:20.191324    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"]
Sep 07 21:08:20.193431 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:08:20.193397    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ebc97d_4abc_4a7e_bbea_a509781a7975.slice/crio-291bbe82d466add52259f990e5df5f8379ae1cf0d976669ac6fb7b40cb075474 WatchSource:0}: Error finding container 291bbe82d466add52259f990e5df5f8379ae1cf0d976669ac6fb7b40cb075474: Status 404 returned error can't find the container with id 291bbe82d466add52259f990e5df5f8379ae1cf0d976669ac6fb7b40cb075474
Sep 07 21:08:20.962822 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:20.962793    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p" event={"ID":"75ebc97d-4abc-4a7e-bbea-a509781a7975","Type":"ContainerStarted","Data":"291bbe82d466add52259f990e5df5f8379ae1cf0d976669ac6fb7b40cb075474"}
Sep 07 21:08:22.974853 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:22.974760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p" event={"ID":"75ebc97d-4abc-4a7e-bbea-a509781a7975","Type":"ContainerStarted","Data":"f53ba0d405a863b6012bface2aae4c157fe7e51f60cb8e38c426e5868ab66c92"}
Sep 07 21:08:22.975238 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:22.974870    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:22.994561 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:22.994511    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p" podStartSLOduration=1.502919045 podStartE2EDuration="3.994494496s" totalImagesPullingTime="2.491575451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:08:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:08:22.991922616 +0000 UTC m=+395.073449969" watchObservedRunningTime="2026-09-07 21:08:22.994494496 +0000 UTC m=+395.076021836"
Sep 07 21:08:23.000205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.000182    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"]
Sep 07 21:08:23.000386 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.000363    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2" podUID="139aa155-b33e-4c81-8b1e-546d5dffdd65" containerName="discovery" containerID="cri-o://4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636" gracePeriod=30
Sep 07 21:08:23.241532 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.241508    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:23.294406 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.294371    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/139aa155-b33e-4c81-8b1e-546d5dffdd65-local-certs\") pod \"139aa155-b33e-4c81-8b1e-546d5dffdd65\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") "
Sep 07 21:08:23.294557 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.294429    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-cacerts\") pod \"139aa155-b33e-4c81-8b1e-546d5dffdd65\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") "
Sep 07 21:08:23.294557 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.294489    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-kubeconfig\") pod \"139aa155-b33e-4c81-8b1e-546d5dffdd65\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") "
Sep 07 21:08:23.294557 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.294518    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4fxs\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-kube-api-access-h4fxs\") pod \"139aa155-b33e-4c81-8b1e-546d5dffdd65\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") "
Sep 07 21:08:23.294557 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.294549    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-dns-cert\") pod \"139aa155-b33e-4c81-8b1e-546d5dffdd65\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") "
Sep 07 21:08:23.294761 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.294603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-ca-configmap\") pod \"139aa155-b33e-4c81-8b1e-546d5dffdd65\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") "
Sep 07 21:08:23.294761 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.294648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-token\") pod \"139aa155-b33e-4c81-8b1e-546d5dffdd65\" (UID: \"139aa155-b33e-4c81-8b1e-546d5dffdd65\") "
Sep 07 21:08:23.295339 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.295277    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "139aa155-b33e-4c81-8b1e-546d5dffdd65" (UID: "139aa155-b33e-4c81-8b1e-546d5dffdd65"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:08:23.297221 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.297121    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "139aa155-b33e-4c81-8b1e-546d5dffdd65" (UID: "139aa155-b33e-4c81-8b1e-546d5dffdd65"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:08:23.297221 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.297189    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/139aa155-b33e-4c81-8b1e-546d5dffdd65-local-certs" (OuterVolumeSpecName: "local-certs") pod "139aa155-b33e-4c81-8b1e-546d5dffdd65" (UID: "139aa155-b33e-4c81-8b1e-546d5dffdd65"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:08:23.297221 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.297198    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-cacerts" (OuterVolumeSpecName: "cacerts") pod "139aa155-b33e-4c81-8b1e-546d5dffdd65" (UID: "139aa155-b33e-4c81-8b1e-546d5dffdd65"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:08:23.297398 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.297241    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "139aa155-b33e-4c81-8b1e-546d5dffdd65" (UID: "139aa155-b33e-4c81-8b1e-546d5dffdd65"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:08:23.297483 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.297460    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-token" (OuterVolumeSpecName: "istio-token") pod "139aa155-b33e-4c81-8b1e-546d5dffdd65" (UID: "139aa155-b33e-4c81-8b1e-546d5dffdd65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:08:23.297693 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.297677    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-kube-api-access-h4fxs" (OuterVolumeSpecName: "kube-api-access-h4fxs") pod "139aa155-b33e-4c81-8b1e-546d5dffdd65" (UID: "139aa155-b33e-4c81-8b1e-546d5dffdd65"). InnerVolumeSpecName "kube-api-access-h4fxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:08:23.395932 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.395903    2565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-cacerts\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:23.395932 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.395926    2565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-kubeconfig\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:23.395932 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.395936    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4fxs\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-kube-api-access-h4fxs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:23.396188 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.395946    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-dns-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:23.396188 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.395954    2565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-csr-ca-configmap\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:23.396188 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.395981    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/139aa155-b33e-4c81-8b1e-546d5dffdd65-istio-token\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:23.396188 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.395990    2565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/139aa155-b33e-4c81-8b1e-546d5dffdd65-local-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:23.985368 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.985270    2565 generic.go:356] "Generic (PLEG): container finished" podID="139aa155-b33e-4c81-8b1e-546d5dffdd65" containerID="4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636" exitCode=0
Sep 07 21:08:23.985368 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.985331    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"
Sep 07 21:08:23.985368 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.985347    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2" event={"ID":"139aa155-b33e-4c81-8b1e-546d5dffdd65","Type":"ContainerDied","Data":"4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636"}
Sep 07 21:08:23.985847 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.985425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2" event={"ID":"139aa155-b33e-4c81-8b1e-546d5dffdd65","Type":"ContainerDied","Data":"8570ff7ca1ce4d849d331772fc4fa58f1ff1628babe44de8cf3db9bfbdf87ec9"}
Sep 07 21:08:23.985847 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.985452    2565 scope.go:117] "RemoveContainer" containerID="4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636"
Sep 07 21:08:23.994827 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.994810    2565 scope.go:117] "RemoveContainer" containerID="4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636"
Sep 07 21:08:23.995255 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:08:23.995236    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636\": container with ID starting with 4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636 not found: ID does not exist" containerID="4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636"
Sep 07 21:08:23.995314 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:23.995262    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636"} err="failed to get container status \"4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636\": rpc error: code = NotFound desc = could not find container \"4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636\": container with ID starting with 4a900260ea6b01b59f4e20d234f6af7c98700c5a054c458b561301aa9176b636 not found: ID does not exist"
Sep 07 21:08:24.006378 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:24.006356    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"]
Sep 07 21:08:24.011401 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:24.011378    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-gfkr2"]
Sep 07 21:08:24.363177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:24.363144    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="139aa155-b33e-4c81-8b1e-546d5dffdd65" path="/var/lib/kubelet/pods/139aa155-b33e-4c81-8b1e-546d5dffdd65/volumes"
Sep 07 21:08:27.684483 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:27.684450    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf56d8966-4mjsf"]
Sep 07 21:08:33.987910 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:33.987875    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-75546db857-h8g6p"
Sep 07 21:08:52.705252 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:52.705194    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cf56d8966-4mjsf" podUID="74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" containerName="console" containerID="cri-o://82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba" gracePeriod=15
Sep 07 21:08:52.952167 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:52.952142    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:08:53.033216 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-oauth-serving-cert\") pod \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") "
Sep 07 21:08:53.033216 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033172    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-trusted-ca-bundle\") pod \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") "
Sep 07 21:08:53.033216 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033189    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-serving-cert\") pod \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") "
Sep 07 21:08:53.033461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033221    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-config\") pod \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") "
Sep 07 21:08:53.033461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033305    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-service-ca\") pod \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") "
Sep 07 21:08:53.033461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033331    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-oauth-config\") pod \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") "
Sep 07 21:08:53.033461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033378    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6779f\" (UniqueName: \"kubernetes.io/projected/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-kube-api-access-6779f\") pod \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\" (UID: \"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb\") "
Sep 07 21:08:53.033728 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033700    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-service-ca" (OuterVolumeSpecName: "service-ca") pod "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" (UID: "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:08:53.033807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033737    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" (UID: "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:08:53.033807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033743    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-config" (OuterVolumeSpecName: "console-config") pod "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" (UID: "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:08:53.033807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.033799    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" (UID: "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:08:53.035626 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.035608    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-kube-api-access-6779f" (OuterVolumeSpecName: "kube-api-access-6779f") pod "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" (UID: "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb"). InnerVolumeSpecName "kube-api-access-6779f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:08:53.035889 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.035863    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" (UID: "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:08:53.035889 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.035873    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" (UID: "74fcbd0a-ad4e-4d24-862a-08fca9ab69fb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:08:53.105506 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.105476    2565 generic.go:356] "Generic (PLEG): container finished" podID="74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" containerID="82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba" exitCode=0
Sep 07 21:08:53.105618 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.105526    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cf56d8966-4mjsf"
Sep 07 21:08:53.105618 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.105534    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf56d8966-4mjsf" event={"ID":"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb","Type":"ContainerDied","Data":"82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba"}
Sep 07 21:08:53.105618 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.105615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cf56d8966-4mjsf" event={"ID":"74fcbd0a-ad4e-4d24-862a-08fca9ab69fb","Type":"ContainerDied","Data":"a65b82544ec434561cd325ed7b663bd0d47966a0cc4ea226d4d9e785911a5c96"}
Sep 07 21:08:53.105733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.105641    2565 scope.go:117] "RemoveContainer" containerID="82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba"
Sep 07 21:08:53.113603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.113584    2565 scope.go:117] "RemoveContainer" containerID="82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba"
Sep 07 21:08:53.113862 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:08:53.113842    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba\": container with ID starting with 82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba not found: ID does not exist" containerID="82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba"
Sep 07 21:08:53.113934 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.113868    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba"} err="failed to get container status \"82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba\": rpc error: code = NotFound desc = could not find container \"82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba\": container with ID starting with 82be5362bf472f952b5a0fc433cda5650806a4dbe079a64153265a2c968ec6ba not found: ID does not exist"
Sep 07 21:08:53.125689 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.125664    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cf56d8966-4mjsf"]
Sep 07 21:08:53.129728 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.129708    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cf56d8966-4mjsf"]
Sep 07 21:08:53.134644 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.134626    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:53.134733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.134649    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-oauth-config\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:53.134733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.134664    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6779f\" (UniqueName: \"kubernetes.io/projected/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-kube-api-access-6779f\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:53.134733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.134678    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-oauth-serving-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:53.134733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.134691    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-trusted-ca-bundle\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:53.134733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.134704    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-serving-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:53.134733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:53.134717    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb-console-config\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:08:54.362654 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:08:54.362621    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" path="/var/lib/kubelet/pods/74fcbd0a-ad4e-4d24-862a-08fca9ab69fb/volumes"
Sep 07 21:10:06.467682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.467646    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"]
Sep 07 21:10:06.468151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.467944    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" containerName="console"
Sep 07 21:10:06.468151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.467958    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" containerName="console"
Sep 07 21:10:06.468151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.467987    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="139aa155-b33e-4c81-8b1e-546d5dffdd65" containerName="discovery"
Sep 07 21:10:06.468151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.467994    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="139aa155-b33e-4c81-8b1e-546d5dffdd65" containerName="discovery"
Sep 07 21:10:06.468151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.468044    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="139aa155-b33e-4c81-8b1e-546d5dffdd65" containerName="discovery"
Sep 07 21:10:06.468151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.468056    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fcbd0a-ad4e-4d24-862a-08fca9ab69fb" containerName="console"
Sep 07 21:10:06.472390 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.472365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.475870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.475843    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 07 21:10:06.475997 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.475847    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lj85v\""
Sep 07 21:10:06.475997 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.475849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 07 21:10:06.475997 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.475919    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 21:10:06.475997 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.475849    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 21:10:06.479052 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.479028    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"]
Sep 07 21:10:06.613407 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.613362    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/685d1787-efd6-4081-8e42-11d63d5e9ad1-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.613604 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.613451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/685d1787-efd6-4081-8e42-11d63d5e9ad1-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.613604 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.613475    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jmtw\" (UniqueName: \"kubernetes.io/projected/685d1787-efd6-4081-8e42-11d63d5e9ad1-kube-api-access-2jmtw\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.714707 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.714659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/685d1787-efd6-4081-8e42-11d63d5e9ad1-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.714707 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.714708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jmtw\" (UniqueName: \"kubernetes.io/projected/685d1787-efd6-4081-8e42-11d63d5e9ad1-kube-api-access-2jmtw\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.714924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.714795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/685d1787-efd6-4081-8e42-11d63d5e9ad1-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.715526 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.715507    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/685d1787-efd6-4081-8e42-11d63d5e9ad1-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.717263 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.717242    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/685d1787-efd6-4081-8e42-11d63d5e9ad1-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.723333 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.723277    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jmtw\" (UniqueName: \"kubernetes.io/projected/685d1787-efd6-4081-8e42-11d63d5e9ad1-kube-api-access-2jmtw\") pod \"kuadrant-console-plugin-598f86468b-wx74s\" (UID: \"685d1787-efd6-4081-8e42-11d63d5e9ad1\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:06.784330 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:06.784298    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"
Sep 07 21:10:07.057941 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:07.057911    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s"]
Sep 07 21:10:07.060346 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:10:07.060318    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod685d1787_efd6_4081_8e42_11d63d5e9ad1.slice/crio-aaa895c16816b61b3a23572c1d195cb83626d8c0c2ced326f76f7ac6e421ca79 WatchSource:0}: Error finding container aaa895c16816b61b3a23572c1d195cb83626d8c0c2ced326f76f7ac6e421ca79: Status 404 returned error can't find the container with id aaa895c16816b61b3a23572c1d195cb83626d8c0c2ced326f76f7ac6e421ca79
Sep 07 21:10:07.405040 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:07.405006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s" event={"ID":"685d1787-efd6-4081-8e42-11d63d5e9ad1","Type":"ContainerStarted","Data":"aaa895c16816b61b3a23572c1d195cb83626d8c0c2ced326f76f7ac6e421ca79"}
Sep 07 21:10:12.429083 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:12.429040    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s" event={"ID":"685d1787-efd6-4081-8e42-11d63d5e9ad1","Type":"ContainerStarted","Data":"0b4c1e12789758291f35667ff5f558af6e4fd61f31d89929768cc526e94b7bb4"}
Sep 07 21:10:12.446559 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:12.446507    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-wx74s" podStartSLOduration=1.8671445759999998 podStartE2EDuration="6.446492453s" totalImagesPullingTime="4.579347877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:10:12.445134945 +0000 UTC m=+504.526662287" watchObservedRunningTime="2026-09-07 21:10:12.446492453 +0000 UTC m=+504.528019794"
Sep 07 21:10:53.115436 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.115395    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:10:53.136878 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.136832    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:10:53.137075 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.136984    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:53.139010 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.138981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:10:53.139406 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.139389    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 07 21:10:53.196106 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.196070    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3383060f-16e7-4691-a7e2-02d67e321513-config-file\") pod \"limitador-limitador-85d9db4d5d-c95tz\" (UID: \"3383060f-16e7-4691-a7e2-02d67e321513\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:53.196273 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.196139    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9txns\" (UniqueName: \"kubernetes.io/projected/3383060f-16e7-4691-a7e2-02d67e321513-kube-api-access-9txns\") pod \"limitador-limitador-85d9db4d5d-c95tz\" (UID: \"3383060f-16e7-4691-a7e2-02d67e321513\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:53.296890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.296855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9txns\" (UniqueName: \"kubernetes.io/projected/3383060f-16e7-4691-a7e2-02d67e321513-kube-api-access-9txns\") pod \"limitador-limitador-85d9db4d5d-c95tz\" (UID: \"3383060f-16e7-4691-a7e2-02d67e321513\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:53.297090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.296912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3383060f-16e7-4691-a7e2-02d67e321513-config-file\") pod \"limitador-limitador-85d9db4d5d-c95tz\" (UID: \"3383060f-16e7-4691-a7e2-02d67e321513\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:53.297576 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.297555    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3383060f-16e7-4691-a7e2-02d67e321513-config-file\") pod \"limitador-limitador-85d9db4d5d-c95tz\" (UID: \"3383060f-16e7-4691-a7e2-02d67e321513\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:53.304514 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.304486    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9txns\" (UniqueName: \"kubernetes.io/projected/3383060f-16e7-4691-a7e2-02d67e321513-kube-api-access-9txns\") pod \"limitador-limitador-85d9db4d5d-c95tz\" (UID: \"3383060f-16e7-4691-a7e2-02d67e321513\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:53.447128 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.447044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:53.479904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.479869    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b54df94b4-k6cqw"]
Sep 07 21:10:53.483493 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.483469    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-k6cqw"
Sep 07 21:10:53.485986 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.485951    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9bpbs\""
Sep 07 21:10:53.490865 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.490844    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-k6cqw"]
Sep 07 21:10:53.601312 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.601277    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjxz6\" (UniqueName: \"kubernetes.io/projected/f65ed193-9389-4357-8e54-d50aef0f189a-kube-api-access-wjxz6\") pod \"authorino-b54df94b4-k6cqw\" (UID: \"f65ed193-9389-4357-8e54-d50aef0f189a\") " pod="kuadrant-system/authorino-b54df94b4-k6cqw"
Sep 07 21:10:53.702088 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.702007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjxz6\" (UniqueName: \"kubernetes.io/projected/f65ed193-9389-4357-8e54-d50aef0f189a-kube-api-access-wjxz6\") pod \"authorino-b54df94b4-k6cqw\" (UID: \"f65ed193-9389-4357-8e54-d50aef0f189a\") " pod="kuadrant-system/authorino-b54df94b4-k6cqw"
Sep 07 21:10:53.710057 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.710029    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjxz6\" (UniqueName: \"kubernetes.io/projected/f65ed193-9389-4357-8e54-d50aef0f189a-kube-api-access-wjxz6\") pod \"authorino-b54df94b4-k6cqw\" (UID: \"f65ed193-9389-4357-8e54-d50aef0f189a\") " pod="kuadrant-system/authorino-b54df94b4-k6cqw"
Sep 07 21:10:53.719326 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.719301    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:10:53.721903 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:10:53.721878    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3383060f_16e7_4691_a7e2_02d67e321513.slice/crio-bcc6c80604d64e588d7c97b5f19ad3fda6b138891c01c0cae739078a68d11b83 WatchSource:0}: Error finding container bcc6c80604d64e588d7c97b5f19ad3fda6b138891c01c0cae739078a68d11b83: Status 404 returned error can't find the container with id bcc6c80604d64e588d7c97b5f19ad3fda6b138891c01c0cae739078a68d11b83
Sep 07 21:10:53.796103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:53.796071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-k6cqw"
Sep 07 21:10:54.061927 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:54.061748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b54df94b4-k6cqw"]
Sep 07 21:10:54.064436 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:10:54.064407    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf65ed193_9389_4357_8e54_d50aef0f189a.slice/crio-e6a55b0dcaaa0ab585002f6814488cf12e82a2aab4bd390902ebe58b8144b8ca WatchSource:0}: Error finding container e6a55b0dcaaa0ab585002f6814488cf12e82a2aab4bd390902ebe58b8144b8ca: Status 404 returned error can't find the container with id e6a55b0dcaaa0ab585002f6814488cf12e82a2aab4bd390902ebe58b8144b8ca
Sep 07 21:10:54.600578 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:54.600538    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz" event={"ID":"3383060f-16e7-4691-a7e2-02d67e321513","Type":"ContainerStarted","Data":"bcc6c80604d64e588d7c97b5f19ad3fda6b138891c01c0cae739078a68d11b83"}
Sep 07 21:10:54.601884 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:54.601851    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-k6cqw" event={"ID":"f65ed193-9389-4357-8e54-d50aef0f189a","Type":"ContainerStarted","Data":"e6a55b0dcaaa0ab585002f6814488cf12e82a2aab4bd390902ebe58b8144b8ca"}
Sep 07 21:10:56.370418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.370379    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-k6cqw"]
Sep 07 21:10:56.613306 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.613255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-k6cqw" event={"ID":"f65ed193-9389-4357-8e54-d50aef0f189a","Type":"ContainerStarted","Data":"dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a"}
Sep 07 21:10:56.613306 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.613263    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-b54df94b4-k6cqw" podUID="f65ed193-9389-4357-8e54-d50aef0f189a" containerName="authorino" containerID="cri-o://dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a" gracePeriod=30
Sep 07 21:10:56.614924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.614895    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz" event={"ID":"3383060f-16e7-4691-a7e2-02d67e321513","Type":"ContainerStarted","Data":"4924f89c67fa3573f6dfea06e621613b54122a5dd4e55d80fb386bd6b70ec0b6"}
Sep 07 21:10:56.615066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.615001    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:10:56.627538 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.627469    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b54df94b4-k6cqw" podStartSLOduration=1.286400098 podStartE2EDuration="3.627454551s" totalImagesPullingTime="2.341054453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:10:56.626537465 +0000 UTC m=+548.708064805" watchObservedRunningTime="2026-09-07 21:10:56.627454551 +0000 UTC m=+548.708981892"
Sep 07 21:10:56.642516 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.642466    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz" podStartSLOduration=0.966527867 podStartE2EDuration="3.642455288s" totalImagesPullingTime="2.675927421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:10:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:10:56.642274217 +0000 UTC m=+548.723801560" watchObservedRunningTime="2026-09-07 21:10:56.642455288 +0000 UTC m=+548.723982641"
Sep 07 21:10:56.863108 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.863083    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-k6cqw"
Sep 07 21:10:56.927390 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.927306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjxz6\" (UniqueName: \"kubernetes.io/projected/f65ed193-9389-4357-8e54-d50aef0f189a-kube-api-access-wjxz6\") pod \"f65ed193-9389-4357-8e54-d50aef0f189a\" (UID: \"f65ed193-9389-4357-8e54-d50aef0f189a\") "
Sep 07 21:10:56.929493 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:56.929470    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f65ed193-9389-4357-8e54-d50aef0f189a-kube-api-access-wjxz6" (OuterVolumeSpecName: "kube-api-access-wjxz6") pod "f65ed193-9389-4357-8e54-d50aef0f189a" (UID: "f65ed193-9389-4357-8e54-d50aef0f189a"). InnerVolumeSpecName "kube-api-access-wjxz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:10:57.028202 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.028163    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjxz6\" (UniqueName: \"kubernetes.io/projected/f65ed193-9389-4357-8e54-d50aef0f189a-kube-api-access-wjxz6\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:10:57.620746 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.620706    2565 generic.go:356] "Generic (PLEG): container finished" podID="f65ed193-9389-4357-8e54-d50aef0f189a" containerID="dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a" exitCode=0
Sep 07 21:10:57.621283 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.620769    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b54df94b4-k6cqw"
Sep 07 21:10:57.621283 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.620806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-k6cqw" event={"ID":"f65ed193-9389-4357-8e54-d50aef0f189a","Type":"ContainerDied","Data":"dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a"}
Sep 07 21:10:57.621283 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.620856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b54df94b4-k6cqw" event={"ID":"f65ed193-9389-4357-8e54-d50aef0f189a","Type":"ContainerDied","Data":"e6a55b0dcaaa0ab585002f6814488cf12e82a2aab4bd390902ebe58b8144b8ca"}
Sep 07 21:10:57.621283 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.620881    2565 scope.go:117] "RemoveContainer" containerID="dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a"
Sep 07 21:10:57.629075 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.629057    2565 scope.go:117] "RemoveContainer" containerID="dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a"
Sep 07 21:10:57.629378 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:10:57.629358    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a\": container with ID starting with dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a not found: ID does not exist" containerID="dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a"
Sep 07 21:10:57.629437 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.629390    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a"} err="failed to get container status \"dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a\": rpc error: code = NotFound desc = could not find container \"dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a\": container with ID starting with dffdc11b20b512b706dc5ad89667d865f4cdf50f3a23ff8e8e5f48fd12cd200a not found: ID does not exist"
Sep 07 21:10:57.640602 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.640577    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-b54df94b4-k6cqw"]
Sep 07 21:10:57.644343 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:57.644321    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-b54df94b4-k6cqw"]
Sep 07 21:10:58.364084 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:10:58.364051    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f65ed193-9389-4357-8e54-d50aef0f189a" path="/var/lib/kubelet/pods/f65ed193-9389-4357-8e54-d50aef0f189a/volumes"
Sep 07 21:11:07.622583 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:07.622552    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"
Sep 07 21:11:21.443311 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.443274    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ktrrd"]
Sep 07 21:11:21.443754 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.443578    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f65ed193-9389-4357-8e54-d50aef0f189a" containerName="authorino"
Sep 07 21:11:21.443754 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.443590    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f65ed193-9389-4357-8e54-d50aef0f189a" containerName="authorino"
Sep 07 21:11:21.443754 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.443645    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f65ed193-9389-4357-8e54-d50aef0f189a" containerName="authorino"
Sep 07 21:11:21.446424 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.446408    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd"
Sep 07 21:11:21.448839 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.448819    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9bpbs\""
Sep 07 21:11:21.453609 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.453379    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ktrrd"]
Sep 07 21:11:21.534199 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.534166    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97djc\" (UniqueName: \"kubernetes.io/projected/80f424f1-d30e-4344-a1fe-3157c4852841-kube-api-access-97djc\") pod \"authorino-5dc7ccb5f-ktrrd\" (UID: \"80f424f1-d30e-4344-a1fe-3157c4852841\") " pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd"
Sep 07 21:11:21.634939 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.634899    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97djc\" (UniqueName: \"kubernetes.io/projected/80f424f1-d30e-4344-a1fe-3157c4852841-kube-api-access-97djc\") pod \"authorino-5dc7ccb5f-ktrrd\" (UID: \"80f424f1-d30e-4344-a1fe-3157c4852841\") " pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd"
Sep 07 21:11:21.643096 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.643065    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97djc\" (UniqueName: \"kubernetes.io/projected/80f424f1-d30e-4344-a1fe-3157c4852841-kube-api-access-97djc\") pod \"authorino-5dc7ccb5f-ktrrd\" (UID: \"80f424f1-d30e-4344-a1fe-3157c4852841\") " pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd"
Sep 07 21:11:21.676123 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.676100    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ktrrd"]
Sep 07 21:11:21.676339 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.676328    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd"
Sep 07 21:11:21.702714 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.702660    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54d8b644f6-zmlgd"]
Sep 07 21:11:21.706149 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.706131    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d8b644f6-zmlgd"
Sep 07 21:11:21.712727 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.712701    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54d8b644f6-zmlgd"]
Sep 07 21:11:21.735453 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.735418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwbd\" (UniqueName: \"kubernetes.io/projected/057f33da-740b-4147-9e2d-9e13962448ff-kube-api-access-skwbd\") pod \"authorino-54d8b644f6-zmlgd\" (UID: \"057f33da-740b-4147-9e2d-9e13962448ff\") " pod="kuadrant-system/authorino-54d8b644f6-zmlgd"
Sep 07 21:11:21.754242 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.754213    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54d8b644f6-zmlgd"]
Sep 07 21:11:21.754496 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:11:21.754459    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-skwbd], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-54d8b644f6-zmlgd" podUID="057f33da-740b-4147-9e2d-9e13962448ff"
Sep 07 21:11:21.836168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.836132    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwbd\" (UniqueName: \"kubernetes.io/projected/057f33da-740b-4147-9e2d-9e13962448ff-kube-api-access-skwbd\") pod \"authorino-54d8b644f6-zmlgd\" (UID: \"057f33da-740b-4147-9e2d-9e13962448ff\") " pod="kuadrant-system/authorino-54d8b644f6-zmlgd"
Sep 07 21:11:21.844289 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.844267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwbd\" (UniqueName: \"kubernetes.io/projected/057f33da-740b-4147-9e2d-9e13962448ff-kube-api-access-skwbd\") pod \"authorino-54d8b644f6-zmlgd\" (UID: \"057f33da-740b-4147-9e2d-9e13962448ff\") " pod="kuadrant-system/authorino-54d8b644f6-zmlgd"
Sep 07 21:11:21.947667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:21.947641    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ktrrd"]
Sep 07 21:11:21.950129 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:11:21.950103    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80f424f1_d30e_4344_a1fe_3157c4852841.slice/crio-70a6ea9cfdb1130e670f6afa211d925a808473ebad054bca6e26552bb19fc8fe WatchSource:0}: Error finding container 70a6ea9cfdb1130e670f6afa211d925a808473ebad054bca6e26552bb19fc8fe: Status 404 returned error can't find the container with id 70a6ea9cfdb1130e670f6afa211d925a808473ebad054bca6e26552bb19fc8fe
Sep 07 21:11:22.729688 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.729608    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd" event={"ID":"80f424f1-d30e-4344-a1fe-3157c4852841","Type":"ContainerStarted","Data":"02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179"}
Sep 07 21:11:22.729688 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.729634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d8b644f6-zmlgd"
Sep 07 21:11:22.729688 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.729634    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd" podUID="80f424f1-d30e-4344-a1fe-3157c4852841" containerName="authorino" containerID="cri-o://02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179" gracePeriod=30
Sep 07 21:11:22.729688 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.729651    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd" event={"ID":"80f424f1-d30e-4344-a1fe-3157c4852841","Type":"ContainerStarted","Data":"70a6ea9cfdb1130e670f6afa211d925a808473ebad054bca6e26552bb19fc8fe"}
Sep 07 21:11:22.737072 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.737053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d8b644f6-zmlgd"
Sep 07 21:11:22.741320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.741295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skwbd\" (UniqueName: \"kubernetes.io/projected/057f33da-740b-4147-9e2d-9e13962448ff-kube-api-access-skwbd\") pod \"057f33da-740b-4147-9e2d-9e13962448ff\" (UID: \"057f33da-740b-4147-9e2d-9e13962448ff\") "
Sep 07 21:11:22.743749 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.743728    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057f33da-740b-4147-9e2d-9e13962448ff-kube-api-access-skwbd" (OuterVolumeSpecName: "kube-api-access-skwbd") pod "057f33da-740b-4147-9e2d-9e13962448ff" (UID: "057f33da-740b-4147-9e2d-9e13962448ff"). InnerVolumeSpecName "kube-api-access-skwbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:11:22.744836 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.744792    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd" podStartSLOduration=1.257495432 podStartE2EDuration="1.744774927s" totalImagesPullingTime="487.279495ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:11:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:11:22.742941837 +0000 UTC m=+574.824469178" watchObservedRunningTime="2026-09-07 21:11:22.744774927 +0000 UTC m=+574.826302269"
Sep 07 21:11:22.842478 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.842451    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skwbd\" (UniqueName: \"kubernetes.io/projected/057f33da-740b-4147-9e2d-9e13962448ff-kube-api-access-skwbd\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:11:22.980723 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:22.980673    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd"
Sep 07 21:11:23.044037 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.044005    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97djc\" (UniqueName: \"kubernetes.io/projected/80f424f1-d30e-4344-a1fe-3157c4852841-kube-api-access-97djc\") pod \"80f424f1-d30e-4344-a1fe-3157c4852841\" (UID: \"80f424f1-d30e-4344-a1fe-3157c4852841\") "
Sep 07 21:11:23.045950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.045930    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f424f1-d30e-4344-a1fe-3157c4852841-kube-api-access-97djc" (OuterVolumeSpecName: "kube-api-access-97djc") pod "80f424f1-d30e-4344-a1fe-3157c4852841" (UID: "80f424f1-d30e-4344-a1fe-3157c4852841"). InnerVolumeSpecName "kube-api-access-97djc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:11:23.145087 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.145064    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97djc\" (UniqueName: \"kubernetes.io/projected/80f424f1-d30e-4344-a1fe-3157c4852841-kube-api-access-97djc\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:11:23.735947 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.735907    2565 generic.go:356] "Generic (PLEG): container finished" podID="80f424f1-d30e-4344-a1fe-3157c4852841" containerID="02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179" exitCode=0
Sep 07 21:11:23.736365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.735948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd" event={"ID":"80f424f1-d30e-4344-a1fe-3157c4852841","Type":"ContainerDied","Data":"02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179"}
Sep 07 21:11:23.736365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.735978    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd"
Sep 07 21:11:23.736365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.736005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dc7ccb5f-ktrrd" event={"ID":"80f424f1-d30e-4344-a1fe-3157c4852841","Type":"ContainerDied","Data":"70a6ea9cfdb1130e670f6afa211d925a808473ebad054bca6e26552bb19fc8fe"}
Sep 07 21:11:23.736365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.736024    2565 scope.go:117] "RemoveContainer" containerID="02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179"
Sep 07 21:11:23.736365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.736189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54d8b644f6-zmlgd"
Sep 07 21:11:23.744780 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.744762    2565 scope.go:117] "RemoveContainer" containerID="02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179"
Sep 07 21:11:23.745070 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:11:23.745047    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179\": container with ID starting with 02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179 not found: ID does not exist" containerID="02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179"
Sep 07 21:11:23.745145 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.745077    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179"} err="failed to get container status \"02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179\": rpc error: code = NotFound desc = could not find container \"02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179\": container with ID starting with 02547bdfc05ee74091bb53e1e24dfac78cf1d7cc15153641f35b4ffca7d29179 not found: ID does not exist"
Sep 07 21:11:23.764665 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.764644    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54d8b644f6-zmlgd"]
Sep 07 21:11:23.767845 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.767822    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54d8b644f6-zmlgd"]
Sep 07 21:11:23.776757 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.776739    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ktrrd"]
Sep 07 21:11:23.779221 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:23.779201    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5dc7ccb5f-ktrrd"]
Sep 07 21:11:24.364256 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:24.364220    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="057f33da-740b-4147-9e2d-9e13962448ff" path="/var/lib/kubelet/pods/057f33da-740b-4147-9e2d-9e13962448ff/volumes"
Sep 07 21:11:24.364445 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:11:24.364432    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f424f1-d30e-4344-a1fe-3157c4852841" path="/var/lib/kubelet/pods/80f424f1-d30e-4344-a1fe-3157c4852841/volumes"
Sep 07 21:12:05.937503 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:05.937469    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:12:27.255687 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:27.255653    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:12:40.960359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:40.960320    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:12:51.164508 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.164473    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"]
Sep 07 21:12:51.164956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.164859    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f424f1-d30e-4344-a1fe-3157c4852841" containerName="authorino"
Sep 07 21:12:51.164956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.164878    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f424f1-d30e-4344-a1fe-3157c4852841" containerName="authorino"
Sep 07 21:12:51.164956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.164947    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f424f1-d30e-4344-a1fe-3157c4852841" containerName="authorino"
Sep 07 21:12:51.167849 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.167830    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.170351 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.170328    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-rhzs9\""
Sep 07 21:12:51.170454 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.170406    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 21:12:51.171508 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.171490    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 07 21:12:51.171508 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.171504    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 21:12:51.180076 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.180056    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"]
Sep 07 21:12:51.239107 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.239080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.239223 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.239121    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.239223 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.239159    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh2rd\" (UniqueName: \"kubernetes.io/projected/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-kube-api-access-bh2rd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.239223 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.239186    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.239223 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.239208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.239354 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.239252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.339955 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.339926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh2rd\" (UniqueName: \"kubernetes.io/projected/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-kube-api-access-bh2rd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.340092 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.339959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.340134 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.340093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.340178 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.340139    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.340220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.340184    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.340270 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.340239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.340522 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.340504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.340588 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.340559    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.340627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.340589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.342138 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.342110    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.342479 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.342461    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.348016 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.347996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh2rd\" (UniqueName: \"kubernetes.io/projected/6be4e6b4-d4f7-4c07-9f52-701aa93836e1-kube-api-access-bh2rd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk\" (UID: \"6be4e6b4-d4f7-4c07-9f52-701aa93836e1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.477907 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.477805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:12:51.750047 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.750024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"]
Sep 07 21:12:51.752520 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:12:51.752494    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be4e6b4_d4f7_4c07_9f52_701aa93836e1.slice/crio-47f88dbe26ef3392d56699500a205312931e003f7d1fac810cfa4c8ea8f105b1 WatchSource:0}: Error finding container 47f88dbe26ef3392d56699500a205312931e003f7d1fac810cfa4c8ea8f105b1: Status 404 returned error can't find the container with id 47f88dbe26ef3392d56699500a205312931e003f7d1fac810cfa4c8ea8f105b1
Sep 07 21:12:51.753876 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:51.753858    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 21:12:52.051253 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:52.051166    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:12:52.110133 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:52.110095    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk" event={"ID":"6be4e6b4-d4f7-4c07-9f52-701aa93836e1","Type":"ContainerStarted","Data":"47f88dbe26ef3392d56699500a205312931e003f7d1fac810cfa4c8ea8f105b1"}
Sep 07 21:12:53.116150 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:53.116114    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk" event={"ID":"6be4e6b4-d4f7-4c07-9f52-701aa93836e1","Type":"ContainerStarted","Data":"19086803d1e4235724a5fc8b201ef564530620bf20d4c1705f66e1a2e6239151"}
Sep 07 21:12:53.133555 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:12:53.133506    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk" podStartSLOduration=1.181980588 podStartE2EDuration="2.133493107s" totalImagesPullingTime="951.512519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:12:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:12:53.131817563 +0000 UTC m=+665.213344903" watchObservedRunningTime="2026-09-07 21:12:53.133493107 +0000 UTC m=+665.215020447"
Sep 07 21:13:01.479006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:01.478954    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:13:01.479466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:01.479054    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:13:01.491231 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:01.491209    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:13:02.167982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:02.167945    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk"
Sep 07 21:13:06.048180 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:06.048135    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:13:13.761454 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:13.761417    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:13:32.262482 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:32.262443    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:13:41.553482 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.553406    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"]
Sep 07 21:13:41.557145 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.557127    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.559595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.559573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 07 21:13:41.564705 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.564678    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"]
Sep 07 21:13:41.651881 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.651847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d3922c-60fe-4726-ae00-74ffe2f164f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.652042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.651898    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.652042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.651988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.652042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.652033    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2zlw\" (UniqueName: \"kubernetes.io/projected/76d3922c-60fe-4726-ae00-74ffe2f164f4-kube-api-access-j2zlw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.652151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.652080    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.652151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.652101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.752773 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.752738    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d3922c-60fe-4726-ae00-74ffe2f164f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.752933 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.752794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.752933 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.752823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.752933 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.752857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2zlw\" (UniqueName: \"kubernetes.io/projected/76d3922c-60fe-4726-ae00-74ffe2f164f4-kube-api-access-j2zlw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.752933 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.752887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.752933 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.752910    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.753384 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.753357    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.753469 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.753401    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.753469 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.753419    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.755182 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.755160    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d3922c-60fe-4726-ae00-74ffe2f164f4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.755445 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.755427    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d3922c-60fe-4726-ae00-74ffe2f164f4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.760558 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.760534    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2zlw\" (UniqueName: \"kubernetes.io/projected/76d3922c-60fe-4726-ae00-74ffe2f164f4-kube-api-access-j2zlw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j\" (UID: \"76d3922c-60fe-4726-ae00-74ffe2f164f4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:41.867236 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:41.867213    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:42.138652 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:42.138622    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"]
Sep 07 21:13:42.140420 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:13:42.140391    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76d3922c_60fe_4726_ae00_74ffe2f164f4.slice/crio-9e99f637eba40fc101afc9055464e34608b00e199dd500807029eca795ed611e WatchSource:0}: Error finding container 9e99f637eba40fc101afc9055464e34608b00e199dd500807029eca795ed611e: Status 404 returned error can't find the container with id 9e99f637eba40fc101afc9055464e34608b00e199dd500807029eca795ed611e
Sep 07 21:13:42.316949 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:42.316907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j" event={"ID":"76d3922c-60fe-4726-ae00-74ffe2f164f4","Type":"ContainerStarted","Data":"9e99f637eba40fc101afc9055464e34608b00e199dd500807029eca795ed611e"}
Sep 07 21:13:43.322771 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:43.322734    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j" event={"ID":"76d3922c-60fe-4726-ae00-74ffe2f164f4","Type":"ContainerStarted","Data":"4553a63da91b208864dabe144524a4b948d312fd99f48542e52d44852c726735"}
Sep 07 21:13:43.338480 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:43.338411    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j" podStartSLOduration=2.160433583 podStartE2EDuration="2.338396808s" totalImagesPullingTime="177.963225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:13:43.337313919 +0000 UTC m=+715.418841265" watchObservedRunningTime="2026-09-07 21:13:43.338396808 +0000 UTC m=+715.419924149"
Sep 07 21:13:44.363416 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:44.363390    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:13:51.867688 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:51.867649    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:51.868230 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:51.867727    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:51.880165 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:51.880143    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:13:52.375257 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:13:52.375227    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j"
Sep 07 21:14:54.923236 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:54.923195    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c9c5f48f5-p9vz7"]
Sep 07 21:14:54.926479 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:54.926456    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7"
Sep 07 21:14:54.928958 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:54.928937    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 07 21:14:54.929723 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:54.929706    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9bpbs\""
Sep 07 21:14:54.936604 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:54.936582    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c9c5f48f5-p9vz7"]
Sep 07 21:14:55.054219 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.054188    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd6tt\" (UniqueName: \"kubernetes.io/projected/b4d8d43e-38d5-4f75-a1eb-d238eff1193a-kube-api-access-hd6tt\") pod \"authorino-6c9c5f48f5-p9vz7\" (UID: \"b4d8d43e-38d5-4f75-a1eb-d238eff1193a\") " pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7"
Sep 07 21:14:55.054400 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.054252    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4d8d43e-38d5-4f75-a1eb-d238eff1193a-tls-cert\") pod \"authorino-6c9c5f48f5-p9vz7\" (UID: \"b4d8d43e-38d5-4f75-a1eb-d238eff1193a\") " pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7"
Sep 07 21:14:55.155275 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.155239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4d8d43e-38d5-4f75-a1eb-d238eff1193a-tls-cert\") pod \"authorino-6c9c5f48f5-p9vz7\" (UID: \"b4d8d43e-38d5-4f75-a1eb-d238eff1193a\") " pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7"
Sep 07 21:14:55.155428 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.155305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd6tt\" (UniqueName: \"kubernetes.io/projected/b4d8d43e-38d5-4f75-a1eb-d238eff1193a-kube-api-access-hd6tt\") pod \"authorino-6c9c5f48f5-p9vz7\" (UID: \"b4d8d43e-38d5-4f75-a1eb-d238eff1193a\") " pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7"
Sep 07 21:14:55.157657 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.157628    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b4d8d43e-38d5-4f75-a1eb-d238eff1193a-tls-cert\") pod \"authorino-6c9c5f48f5-p9vz7\" (UID: \"b4d8d43e-38d5-4f75-a1eb-d238eff1193a\") " pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7"
Sep 07 21:14:55.162620 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.162601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd6tt\" (UniqueName: \"kubernetes.io/projected/b4d8d43e-38d5-4f75-a1eb-d238eff1193a-kube-api-access-hd6tt\") pod \"authorino-6c9c5f48f5-p9vz7\" (UID: \"b4d8d43e-38d5-4f75-a1eb-d238eff1193a\") " pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7"
Sep 07 21:14:55.235651 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.235579    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7"
Sep 07 21:14:55.503899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.503872    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c9c5f48f5-p9vz7"]
Sep 07 21:14:55.506273 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:14:55.506247    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4d8d43e_38d5_4f75_a1eb_d238eff1193a.slice/crio-390177c76634946d818d54bff7f3cf61c150db0c9655999be170f274fd6435fc WatchSource:0}: Error finding container 390177c76634946d818d54bff7f3cf61c150db0c9655999be170f274fd6435fc: Status 404 returned error can't find the container with id 390177c76634946d818d54bff7f3cf61c150db0c9655999be170f274fd6435fc
Sep 07 21:14:55.616290 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:55.616257    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7" event={"ID":"b4d8d43e-38d5-4f75-a1eb-d238eff1193a","Type":"ContainerStarted","Data":"390177c76634946d818d54bff7f3cf61c150db0c9655999be170f274fd6435fc"}
Sep 07 21:14:58.631302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:58.631273    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7" event={"ID":"b4d8d43e-38d5-4f75-a1eb-d238eff1193a","Type":"ContainerStarted","Data":"0193b320f517124bdc7ac5e45112628646f4b06948be325b7623f233083d9ca9"}
Sep 07 21:14:58.645170 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:14:58.645121    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6c9c5f48f5-p9vz7" podStartSLOduration=1.591370243 podStartE2EDuration="4.645105281s" totalImagesPullingTime="3.053735038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:14:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:14:58.644822154 +0000 UTC m=+790.726349589" watchObservedRunningTime="2026-09-07 21:14:58.645105281 +0000 UTC m=+790.726632623"
Sep 07 21:15:17.670821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:17.670733    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:17.674005 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:17.673988    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:17.676276 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:17.676251    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6qlgr\""
Sep 07 21:15:17.680142 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:17.680121    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:17.850543 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:17.850513    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqhf2\" (UniqueName: \"kubernetes.io/projected/50c6abaf-7916-4217-9e49-407d1e703d5e-kube-api-access-lqhf2\") pod \"test-curl-7505-140136947205808\" (UID: \"50c6abaf-7916-4217-9e49-407d1e703d5e\") " pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:17.951944 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:17.951855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqhf2\" (UniqueName: \"kubernetes.io/projected/50c6abaf-7916-4217-9e49-407d1e703d5e-kube-api-access-lqhf2\") pod \"test-curl-7505-140136947205808\" (UID: \"50c6abaf-7916-4217-9e49-407d1e703d5e\") " pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:17.959345 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:17.959325    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqhf2\" (UniqueName: \"kubernetes.io/projected/50c6abaf-7916-4217-9e49-407d1e703d5e-kube-api-access-lqhf2\") pod \"test-curl-7505-140136947205808\" (UID: \"50c6abaf-7916-4217-9e49-407d1e703d5e\") " pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:17.983947 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:17.983910    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:18.249471 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:18.249444    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:18.251845 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:15:18.251819    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50c6abaf_7916_4217_9e49_407d1e703d5e.slice/crio-10ab7cdb161ba3fbf49c8ff2bac753dba5d28e5fa26a37e0945d022ede32f256 WatchSource:0}: Error finding container 10ab7cdb161ba3fbf49c8ff2bac753dba5d28e5fa26a37e0945d022ede32f256: Status 404 returned error can't find the container with id 10ab7cdb161ba3fbf49c8ff2bac753dba5d28e5fa26a37e0945d022ede32f256
Sep 07 21:15:18.718807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:18.718775    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7505-140136947205808" event={"ID":"50c6abaf-7916-4217-9e49-407d1e703d5e","Type":"ContainerStarted","Data":"10ab7cdb161ba3fbf49c8ff2bac753dba5d28e5fa26a37e0945d022ede32f256"}
Sep 07 21:15:19.956347 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:19.956305    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:15:21.736320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:21.736235    2565 generic.go:356] "Generic (PLEG): container finished" podID="50c6abaf-7916-4217-9e49-407d1e703d5e" containerID="169389720b18e386ada850aa471c746c2dbb900313337a95252795c4ff4802af" exitCode=0
Sep 07 21:15:21.736320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:21.736269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7505-140136947205808" event={"ID":"50c6abaf-7916-4217-9e49-407d1e703d5e","Type":"ContainerDied","Data":"169389720b18e386ada850aa471c746c2dbb900313337a95252795c4ff4802af"}
Sep 07 21:15:22.867027 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:22.867001    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:22.891905 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:22.891877    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqhf2\" (UniqueName: \"kubernetes.io/projected/50c6abaf-7916-4217-9e49-407d1e703d5e-kube-api-access-lqhf2\") pod \"50c6abaf-7916-4217-9e49-407d1e703d5e\" (UID: \"50c6abaf-7916-4217-9e49-407d1e703d5e\") "
Sep 07 21:15:22.894034 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:22.894002    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50c6abaf-7916-4217-9e49-407d1e703d5e-kube-api-access-lqhf2" (OuterVolumeSpecName: "kube-api-access-lqhf2") pod "50c6abaf-7916-4217-9e49-407d1e703d5e" (UID: "50c6abaf-7916-4217-9e49-407d1e703d5e"). InnerVolumeSpecName "kube-api-access-lqhf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:15:22.900430 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:22.900410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7505-140136947205808_50c6abaf-7916-4217-9e49-407d1e703d5e/test-curl-7505-140136947205808/0.log"
Sep 07 21:15:22.920016 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:22.919984    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:22.921601 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:22.921578    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:22.993254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:22.993196    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqhf2\" (UniqueName: \"kubernetes.io/projected/50c6abaf-7916-4217-9e49-407d1e703d5e-kube-api-access-lqhf2\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:15:23.038014 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.037985    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:23.038306 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.038294    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50c6abaf-7916-4217-9e49-407d1e703d5e" containerName="test-curl-7505-140136947205808"
Sep 07 21:15:23.038353 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.038308    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="50c6abaf-7916-4217-9e49-407d1e703d5e" containerName="test-curl-7505-140136947205808"
Sep 07 21:15:23.038386 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.038380    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="50c6abaf-7916-4217-9e49-407d1e703d5e" containerName="test-curl-7505-140136947205808"
Sep 07 21:15:23.041142 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.041128    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:23.045485 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.045460    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:23.094161 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.094135    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq82c\" (UniqueName: \"kubernetes.io/projected/7c351979-424b-4928-a455-130ee41a41dd-kube-api-access-gq82c\") pod \"test-curl-7505-140136947205808\" (UID: \"7c351979-424b-4928-a455-130ee41a41dd\") " pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:23.195395 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.195362    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq82c\" (UniqueName: \"kubernetes.io/projected/7c351979-424b-4928-a455-130ee41a41dd-kube-api-access-gq82c\") pod \"test-curl-7505-140136947205808\" (UID: \"7c351979-424b-4928-a455-130ee41a41dd\") " pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:23.202959 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.202937    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq82c\" (UniqueName: \"kubernetes.io/projected/7c351979-424b-4928-a455-130ee41a41dd-kube-api-access-gq82c\") pod \"test-curl-7505-140136947205808\" (UID: \"7c351979-424b-4928-a455-130ee41a41dd\") " pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:23.351490 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.351461    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:23.622059 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.622033    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:23.624567 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:15:23.624539    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c351979_424b_4928_a455_130ee41a41dd.slice/crio-e799d46f2c60b29bfcd5899b5eca341281f34950da95c30834ac420873d3bb2e WatchSource:0}: Error finding container e799d46f2c60b29bfcd5899b5eca341281f34950da95c30834ac420873d3bb2e: Status 404 returned error can't find the container with id e799d46f2c60b29bfcd5899b5eca341281f34950da95c30834ac420873d3bb2e
Sep 07 21:15:23.747032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.747000    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:23.747032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.747017    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10ab7cdb161ba3fbf49c8ff2bac753dba5d28e5fa26a37e0945d022ede32f256"
Sep 07 21:15:23.748253 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.748233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7505-140136947205808" event={"ID":"7c351979-424b-4928-a455-130ee41a41dd","Type":"ContainerStarted","Data":"e799d46f2c60b29bfcd5899b5eca341281f34950da95c30834ac420873d3bb2e"}
Sep 07 21:15:23.758061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:23.758026    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7505-140136947205808" oldPodUID="50c6abaf-7916-4217-9e49-407d1e703d5e" podUID="7c351979-424b-4928-a455-130ee41a41dd"
Sep 07 21:15:24.363518 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:24.363487    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50c6abaf-7916-4217-9e49-407d1e703d5e" path="/var/lib/kubelet/pods/50c6abaf-7916-4217-9e49-407d1e703d5e/volumes"
Sep 07 21:15:25.764299 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:25.764256    2565 generic.go:356] "Generic (PLEG): container finished" podID="7c351979-424b-4928-a455-130ee41a41dd" containerID="0b5692b850c4fe6661005137d1cde4d2c4fde7a370f727ba1a64b078f1ced220" exitCode=0
Sep 07 21:15:25.764695 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:25.764339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7505-140136947205808" event={"ID":"7c351979-424b-4928-a455-130ee41a41dd","Type":"ContainerDied","Data":"0b5692b850c4fe6661005137d1cde4d2c4fde7a370f727ba1a64b078f1ced220"}
Sep 07 21:15:26.907779 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:26.907752    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:26.929130 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:26.929107    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gq82c\" (UniqueName: \"kubernetes.io/projected/7c351979-424b-4928-a455-130ee41a41dd-kube-api-access-gq82c\") pod \"7c351979-424b-4928-a455-130ee41a41dd\" (UID: \"7c351979-424b-4928-a455-130ee41a41dd\") "
Sep 07 21:15:26.931298 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:26.931269    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c351979-424b-4928-a455-130ee41a41dd-kube-api-access-gq82c" (OuterVolumeSpecName: "kube-api-access-gq82c") pod "7c351979-424b-4928-a455-130ee41a41dd" (UID: "7c351979-424b-4928-a455-130ee41a41dd"). InnerVolumeSpecName "kube-api-access-gq82c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:15:26.941494 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:26.941476    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7505-140136947205808_7c351979-424b-4928-a455-130ee41a41dd/test-curl-7505-140136947205808/0.log"
Sep 07 21:15:26.978128 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:26.978098    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:26.987450 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:26.987426    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7505-140136947205808"]
Sep 07 21:15:27.030074 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.029991    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq82c\" (UniqueName: \"kubernetes.io/projected/7c351979-424b-4928-a455-130ee41a41dd-kube-api-access-gq82c\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:15:27.774062 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.774036    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e799d46f2c60b29bfcd5899b5eca341281f34950da95c30834ac420873d3bb2e"
Sep 07 21:15:27.774204 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.774089    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:27.783971 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:15:27.783933    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7505-140136947205808\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="7c351979-424b-4928-a455-130ee41a41dd" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:27.845082 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.845044    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"]
Sep 07 21:15:27.845418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.845404    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c351979-424b-4928-a455-130ee41a41dd" containerName="test-curl-7505-140136947205808"
Sep 07 21:15:27.845486 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.845419    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c351979-424b-4928-a455-130ee41a41dd" containerName="test-curl-7505-140136947205808"
Sep 07 21:15:27.845486 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.845478    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c351979-424b-4928-a455-130ee41a41dd" containerName="test-curl-7505-140136947205808"
Sep 07 21:15:27.848770 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.848749    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:27.851103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.851083    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-9e7d7ca5-kserve-self-signed-certs\""
Sep 07 21:15:27.858671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.858651    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"]
Sep 07 21:15:27.875797 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:15:27.875766    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7505-140136947205808\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="7c351979-424b-4928-a455-130ee41a41dd" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:27.937948 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.937910    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c7sj\" (UniqueName: \"kubernetes.io/projected/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-kube-api-access-8c7sj\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:27.938372 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.937956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-dshm\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:27.938372 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.938036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-model-cache\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:27.938372 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.938098    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-home\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:27.938372 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.938123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tmp-dir\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:27.938372 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:27.938154    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tls-certs\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.039344 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.039263    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-dshm\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.039344 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.039300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-model-cache\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.039547 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.039348    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-home\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.039547 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.039382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tmp-dir\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.039547 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.039408    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tls-certs\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.039547 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.039481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c7sj\" (UniqueName: \"kubernetes.io/projected/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-kube-api-access-8c7sj\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.039866 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.039832    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-home\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.040016 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.039988    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tmp-dir\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.040090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.040012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-model-cache\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.041759 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.041589    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-dshm\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.042164 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.042143    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tls-certs\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.049062 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.049041    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c7sj\" (UniqueName: \"kubernetes.io/projected/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-kube-api-access-8c7sj\") pod \"e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") " pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.157865 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.157826    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:28.362161 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:15:28.362117    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7505-140136947205808\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="7c351979-424b-4928-a455-130ee41a41dd" pod="opendatahub/test-curl-7505-140136947205808"
Sep 07 21:15:28.364950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.364920    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c351979-424b-4928-a455-130ee41a41dd" path="/var/lib/kubelet/pods/7c351979-424b-4928-a455-130ee41a41dd/volumes"
Sep 07 21:15:28.431816 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.431788    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"]
Sep 07 21:15:28.433572 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:15:28.433546    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2601f5b_b506_4d2b_9ca5_7d17ddacd654.slice/crio-ec1d39eae00996bf429532346d172bd855c5b65fce99d6ec50d21642ada0d2ca WatchSource:0}: Error finding container ec1d39eae00996bf429532346d172bd855c5b65fce99d6ec50d21642ada0d2ca: Status 404 returned error can't find the container with id ec1d39eae00996bf429532346d172bd855c5b65fce99d6ec50d21642ada0d2ca
Sep 07 21:15:28.780864 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.780782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s" event={"ID":"f2601f5b-b506-4d2b-9ca5-7d17ddacd654","Type":"ContainerStarted","Data":"d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be"}
Sep 07 21:15:28.780864 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.780823    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s" event={"ID":"f2601f5b-b506-4d2b-9ca5-7d17ddacd654","Type":"ContainerStarted","Data":"ec1d39eae00996bf429532346d172bd855c5b65fce99d6ec50d21642ada0d2ca"}
Sep 07 21:15:28.797035 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:28.796986    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s" podStartSLOduration=1.796956605 podStartE2EDuration="1.796956605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:15:28.795085801 +0000 UTC m=+820.876613143" watchObservedRunningTime="2026-09-07 21:15:28.796956605 +0000 UTC m=+820.878483946"
Sep 07 21:15:34.370021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:34.369988    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:15:38.158433 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:38.158377    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:38.158894 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:38.158598    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:38.171154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:38.171126    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:38.834946 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:38.834914    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:15:44.650074 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.650039    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"]
Sep 07 21:15:44.653554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.653532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.655672 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.655651    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3102b2-openshift-default-dockercfg-gzhbj\""
Sep 07 21:15:44.655795 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.655692    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 21:15:44.663900 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.663875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"]
Sep 07 21:15:44.785168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785131    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-socket\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.785373 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785181    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-certs\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.785373 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-credential-socket\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.785373 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-podinfo\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.785373 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785325    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-token\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.785373 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2424e7f-f62f-4680-aaee-8aa767988e45-istiod-ca-cert\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.785638 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-data\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.785638 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-envoy\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.785638 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.785564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8cj\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-kube-api-access-2s8cj\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.886564 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886532    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-envoy\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.886724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886570    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8cj\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-kube-api-access-2s8cj\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.886724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886608    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-socket\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.886724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-certs\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.886724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-credential-socket\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.886724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-podinfo\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.886724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886679    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-token\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.886724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886705    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2424e7f-f62f-4680-aaee-8aa767988e45-istiod-ca-cert\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.887084 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.886785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-data\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.887164 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.887135    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-credential-socket\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.887257 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.887162    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-socket\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.887257 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.887207    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-data\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.887347 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.887267    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-certs\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.887817 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.887791    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2424e7f-f62f-4680-aaee-8aa767988e45-istiod-ca-cert\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.888874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.888854    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-envoy\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.889222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.889200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-podinfo\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.895421 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.895391    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-token\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.895508 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.895470    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8cj\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-kube-api-access-2s8cj\") pod \"e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") " pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:44.964991 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:44.964888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:45.241609 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:45.241505    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"]
Sep 07 21:15:45.244553 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:15:45.244528    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2424e7f_f62f_4680_aaee_8aa767988e45.slice/crio-cdf342782be0c2a031da77c2eace1cc871e8026eafa56aff5324e63a0a1ad0a8 WatchSource:0}: Error finding container cdf342782be0c2a031da77c2eace1cc871e8026eafa56aff5324e63a0a1ad0a8: Status 404 returned error can't find the container with id cdf342782be0c2a031da77c2eace1cc871e8026eafa56aff5324e63a0a1ad0a8
Sep 07 21:15:45.858558 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:45.858514    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" event={"ID":"b2424e7f-f62f-4680-aaee-8aa767988e45","Type":"ContainerStarted","Data":"cdf342782be0c2a031da77c2eace1cc871e8026eafa56aff5324e63a0a1ad0a8"}
Sep 07 21:15:47.516800 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:47.516771    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:15:47.517066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:47.516838    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:15:47.517066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:47.516867    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:15:47.869592 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:47.869558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" event={"ID":"b2424e7f-f62f-4680-aaee-8aa767988e45","Type":"ContainerStarted","Data":"35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e"}
Sep 07 21:15:47.890527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:47.890481    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podStartSLOduration=1.621090085 podStartE2EDuration="3.890464207s" totalImagesPullingTime="2.269374122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:15:47.888368329 +0000 UTC m=+839.969895671" watchObservedRunningTime="2026-09-07 21:15:47.890464207 +0000 UTC m=+839.971991547"
Sep 07 21:15:47.965943 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:47.965912    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:48.970836 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:48.970807    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:49.877606 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:49.877574    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:49.878644 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:49.878626    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:15:52.095165 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.095122    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"]
Sep 07 21:15:52.103565 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.103539    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"
Sep 07 21:15:52.105867 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.105843    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:15:52.106056 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.105954    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"]
Sep 07 21:15:52.106133 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.105990    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:15:52.253172 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.253141    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea5fb7-f396-4999-8d1f-63dfd468bfbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2\" (UID: \"49ea5fb7-f396-4999-8d1f-63dfd468bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"
Sep 07 21:15:52.353807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.353726    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea5fb7-f396-4999-8d1f-63dfd468bfbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2\" (UID: \"49ea5fb7-f396-4999-8d1f-63dfd468bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"
Sep 07 21:15:52.354374 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.354355    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea5fb7-f396-4999-8d1f-63dfd468bfbe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2\" (UID: \"49ea5fb7-f396-4999-8d1f-63dfd468bfbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"
Sep 07 21:15:52.414879 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.414832    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"
Sep 07 21:15:52.686289 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.686207    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"]
Sep 07 21:15:52.688945 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:15:52.688921    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ea5fb7_f396_4999_8d1f_63dfd468bfbe.slice/crio-c45f4ff6d6af02096ca756b6c6483b123260cc7b71644941612c6608d31f52c4 WatchSource:0}: Error finding container c45f4ff6d6af02096ca756b6c6483b123260cc7b71644941612c6608d31f52c4: Status 404 returned error can't find the container with id c45f4ff6d6af02096ca756b6c6483b123260cc7b71644941612c6608d31f52c4
Sep 07 21:15:52.892624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:52.892591    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2" event={"ID":"49ea5fb7-f396-4999-8d1f-63dfd468bfbe","Type":"ContainerStarted","Data":"c45f4ff6d6af02096ca756b6c6483b123260cc7b71644941612c6608d31f52c4"}
Sep 07 21:15:54.904403 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:54.904334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2" event={"ID":"49ea5fb7-f396-4999-8d1f-63dfd468bfbe","Type":"ContainerStarted","Data":"6faf1653f20125610bbfabbed36f532182364e9cff0aa95ee5c3fc0aa94f3299"}
Sep 07 21:15:54.919521 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:54.919469    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2" podStartSLOduration=0.9848716 podStartE2EDuration="2.919455268s" totalImagesPullingTime="1.934583668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:15:54.91831421 +0000 UTC m=+846.999841552" watchObservedRunningTime="2026-09-07 21:15:54.919455268 +0000 UTC m=+847.000982609"
Sep 07 21:15:55.910523 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:55.910491    2565 generic.go:356] "Generic (PLEG): container finished" podID="49ea5fb7-f396-4999-8d1f-63dfd468bfbe" containerID="6faf1653f20125610bbfabbed36f532182364e9cff0aa95ee5c3fc0aa94f3299" exitCode=0
Sep 07 21:15:55.910927 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:55.910580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2" event={"ID":"49ea5fb7-f396-4999-8d1f-63dfd468bfbe","Type":"ContainerDied","Data":"6faf1653f20125610bbfabbed36f532182364e9cff0aa95ee5c3fc0aa94f3299"}
Sep 07 21:15:56.661447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:56.661414    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:15:57.057224 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:57.057197    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"
Sep 07 21:15:57.198426 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:57.198347    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea5fb7-f396-4999-8d1f-63dfd468bfbe-maas-api-service-ca\") pod \"49ea5fb7-f396-4999-8d1f-63dfd468bfbe\" (UID: \"49ea5fb7-f396-4999-8d1f-63dfd468bfbe\") "
Sep 07 21:15:57.198688 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:57.198666    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ea5fb7-f396-4999-8d1f-63dfd468bfbe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "49ea5fb7-f396-4999-8d1f-63dfd468bfbe" (UID: "49ea5fb7-f396-4999-8d1f-63dfd468bfbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:15:57.299459 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:57.299413    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49ea5fb7-f396-4999-8d1f-63dfd468bfbe-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:15:57.921702 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:57.921669    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2" event={"ID":"49ea5fb7-f396-4999-8d1f-63dfd468bfbe","Type":"ContainerDied","Data":"c45f4ff6d6af02096ca756b6c6483b123260cc7b71644941612c6608d31f52c4"}
Sep 07 21:15:57.921702 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:57.921701    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"
Sep 07 21:15:57.921897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:15:57.921716    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c45f4ff6d6af02096ca756b6c6483b123260cc7b71644941612c6608d31f52c4"
Sep 07 21:16:05.919133 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:05.919092    2565 patch_prober.go:28] interesting pod/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 21:16:05.919523 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:05.919182    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 21:16:07.970113 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:07.970080    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerID="35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e" exitCode=0
Sep 07 21:16:07.970524 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:07.970157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" event={"ID":"b2424e7f-f62f-4680-aaee-8aa767988e45","Type":"ContainerDied","Data":"35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e"}
Sep 07 21:16:07.970606 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:07.970588    2565 scope.go:117] "RemoveContainer" containerID="35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e"
Sep 07 21:16:07.971632 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:07.971600    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:07.971686 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:07.971672    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:07.971720 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:07.971703    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:08.781979 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:08.781919    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"]
Sep 07 21:16:08.786807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:08.786781    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-28fa6ddd-994t2"]
Sep 07 21:16:08.989860 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:08.989831    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" event={"ID":"b2424e7f-f62f-4680-aaee-8aa767988e45","Type":"ContainerStarted","Data":"7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2"}
Sep 07 21:16:09.891181 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.891137    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"]
Sep 07 21:16:09.891679 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.891656    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49ea5fb7-f396-4999-8d1f-63dfd468bfbe" containerName="revoke-keys"
Sep 07 21:16:09.891679 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.891680    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="49ea5fb7-f396-4999-8d1f-63dfd468bfbe" containerName="revoke-keys"
Sep 07 21:16:09.891854 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.891772    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="49ea5fb7-f396-4999-8d1f-63dfd468bfbe" containerName="revoke-keys"
Sep 07 21:16:09.895189 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.895169    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.897557 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.897538    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f86b6ac3-gw-openshift-default-dockercfg-ssj8f\""
Sep 07 21:16:09.901222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.901197    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"]
Sep 07 21:16:09.906321 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-envoy\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.906321 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906325    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-data\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.906491 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-credential-socket\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.906491 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906382    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-token\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.906491 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906459    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-socket\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.906659 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906510    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8qr2\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-kube-api-access-c8qr2\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.906659 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906543    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-podinfo\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.906768 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6365827-6045-41c2-a858-a2b06a8cf7eb-istiod-ca-cert\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.906850 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.906826    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-certs\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:09.972987 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.970346    2565 patch_prober.go:28] interesting pod/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 21:16:09.972987 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.970414    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 21:16:09.972987 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:09.970680    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:16:10.008164 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008136    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-certs\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008164 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-envoy\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-data\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008212    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-credential-socket\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-token\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008410    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-socket\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008441    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8qr2\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-kube-api-access-c8qr2\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008520    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-podinfo\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008588    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-certs\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008629    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-data\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008646    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-credential-socket\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.008995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.008774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6365827-6045-41c2-a858-a2b06a8cf7eb-istiod-ca-cert\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.009194 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.009177    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-socket\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.009589 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.009567    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6365827-6045-41c2-a858-a2b06a8cf7eb-istiod-ca-cert\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.010658 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.010633    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-envoy\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.011021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.011005    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-podinfo\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.015935 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.015917    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8qr2\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-kube-api-access-c8qr2\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.016076 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.016058    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-token\") pod \"e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") " pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.207291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.207205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:10.387254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.387211    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49ea5fb7-f396-4999-8d1f-63dfd468bfbe" path="/var/lib/kubelet/pods/49ea5fb7-f396-4999-8d1f-63dfd468bfbe/volumes"
Sep 07 21:16:10.492213 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.492128    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"]
Sep 07 21:16:10.495471 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:16:10.495446    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6365827_6045_41c2_a858_a2b06a8cf7eb.slice/crio-d0f1979c3c286dceac5dcf3777e54a9afc906a34d309d6e91f7db6f0be44aa57 WatchSource:0}: Error finding container d0f1979c3c286dceac5dcf3777e54a9afc906a34d309d6e91f7db6f0be44aa57: Status 404 returned error can't find the container with id d0f1979c3c286dceac5dcf3777e54a9afc906a34d309d6e91f7db6f0be44aa57
Sep 07 21:16:10.497792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.497761    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:10.497851 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.497825    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:10.497890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.497853    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:10.965493 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.965445    2565 patch_prober.go:28] interesting pod/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 21:16:10.965695 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:10.965518    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 21:16:11.000902 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:11.000863    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" event={"ID":"b6365827-6045-41c2-a858-a2b06a8cf7eb","Type":"ContainerStarted","Data":"4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb"}
Sep 07 21:16:11.001095 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:11.000920    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" event={"ID":"b6365827-6045-41c2-a858-a2b06a8cf7eb","Type":"ContainerStarted","Data":"d0f1979c3c286dceac5dcf3777e54a9afc906a34d309d6e91f7db6f0be44aa57"}
Sep 07 21:16:11.021514 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:11.021464    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podStartSLOduration=2.021449525 podStartE2EDuration="2.021449525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:16:11.019787456 +0000 UTC m=+863.101314822" watchObservedRunningTime="2026-09-07 21:16:11.021449525 +0000 UTC m=+863.102976932"
Sep 07 21:16:11.208098 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:11.208058    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:11.213208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:11.213184    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:11.965683 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:11.965638    2565 patch_prober.go:28] interesting pod/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 21:16:11.965901 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:11.965702    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 21:16:12.005577 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:12.005547    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:12.006568 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:12.006552    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:12.966421 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:12.966372    2565 patch_prober.go:28] interesting pod/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 21:16:12.967040 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:12.967002    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 21:16:13.418049 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.418008    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"]
Sep 07 21:16:13.424242 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.424221    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:13.426922 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.426896    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f86b6ac3\""
Sep 07 21:16:13.427067 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.426896    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f86b6ac3-dockercfg-4tbjc\""
Sep 07 21:16:13.431088 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.431062    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"]
Sep 07 21:16:13.433649 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.433631    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"]
Sep 07 21:16:13.435622 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.435605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:13.444061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.444036    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sdbv\" (UniqueName: \"kubernetes.io/projected/9d42deef-a708-4caf-bcd2-7b170e361a35-kube-api-access-6sdbv\") pod \"payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt\" (UID: \"9d42deef-a708-4caf-bcd2-7b170e361a35\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:13.444061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.444070    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d42deef-a708-4caf-bcd2-7b170e361a35-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt\" (UID: \"9d42deef-a708-4caf-bcd2-7b170e361a35\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:13.446030 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.446009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"]
Sep 07 21:16:13.544771 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.544734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2cn\" (UniqueName: \"kubernetes.io/projected/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-kube-api-access-rp2cn\") pod \"payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf\" (UID: \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\") " pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:13.544929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.544833    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf\" (UID: \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\") " pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:13.544929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.544860    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sdbv\" (UniqueName: \"kubernetes.io/projected/9d42deef-a708-4caf-bcd2-7b170e361a35-kube-api-access-6sdbv\") pod \"payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt\" (UID: \"9d42deef-a708-4caf-bcd2-7b170e361a35\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:13.544929 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.544883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d42deef-a708-4caf-bcd2-7b170e361a35-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt\" (UID: \"9d42deef-a708-4caf-bcd2-7b170e361a35\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:13.545617 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.545598    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d42deef-a708-4caf-bcd2-7b170e361a35-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt\" (UID: \"9d42deef-a708-4caf-bcd2-7b170e361a35\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:13.553122 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.553101    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sdbv\" (UniqueName: \"kubernetes.io/projected/9d42deef-a708-4caf-bcd2-7b170e361a35-kube-api-access-6sdbv\") pod \"payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt\" (UID: \"9d42deef-a708-4caf-bcd2-7b170e361a35\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:13.646343 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.646300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2cn\" (UniqueName: \"kubernetes.io/projected/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-kube-api-access-rp2cn\") pod \"payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf\" (UID: \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\") " pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:13.646541 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.646399    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf\" (UID: \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\") " pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:13.647078 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.647059    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf\" (UID: \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\") " pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:13.654203 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.654182    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2cn\" (UniqueName: \"kubernetes.io/projected/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-kube-api-access-rp2cn\") pod \"payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf\" (UID: \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\") " pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:13.736095 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.735999    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:13.744987 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.744955    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:13.966463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.966431    2565 patch_prober.go:28] interesting pod/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Sep 07 21:16:13.966845 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:13.966484    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Sep 07 21:16:14.033657 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:14.033576    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"]
Sep 07 21:16:14.036418 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:16:14.036388    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d42deef_a708_4caf_bcd2_7b170e361a35.slice/crio-a33cf7eeb4e43022a6d05ee1ec65a4b25a864289e036a428c7ea22fda381914a WatchSource:0}: Error finding container a33cf7eeb4e43022a6d05ee1ec65a4b25a864289e036a428c7ea22fda381914a: Status 404 returned error can't find the container with id a33cf7eeb4e43022a6d05ee1ec65a4b25a864289e036a428c7ea22fda381914a
Sep 07 21:16:14.045126 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:14.045098    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"]
Sep 07 21:16:14.047588 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:16:14.047561    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c294631_e3ae_4689_9a3f_4a3b7cc8ab66.slice/crio-353b93ea3df2ab943af4e1adef291ce8e477fb022c5c34f18bb8f8c16318cf5d WatchSource:0}: Error finding container 353b93ea3df2ab943af4e1adef291ce8e477fb022c5c34f18bb8f8c16318cf5d: Status 404 returned error can't find the container with id 353b93ea3df2ab943af4e1adef291ce8e477fb022c5c34f18bb8f8c16318cf5d
Sep 07 21:16:14.972186 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:14.972154    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:16:14.973993 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:14.972822    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:16:14.974259 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:14.974205    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:16:15.026836 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:15.026798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf" event={"ID":"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66","Type":"ContainerStarted","Data":"353b93ea3df2ab943af4e1adef291ce8e477fb022c5c34f18bb8f8c16318cf5d"}
Sep 07 21:16:15.031212 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:15.031102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt" event={"ID":"9d42deef-a708-4caf-bcd2-7b170e361a35","Type":"ContainerStarted","Data":"a33cf7eeb4e43022a6d05ee1ec65a4b25a864289e036a428c7ea22fda381914a"}
Sep 07 21:16:17.042096 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:17.042056    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf" event={"ID":"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66","Type":"ContainerStarted","Data":"6e3aff369caeb7a91b44f3f476fa4a6877f9e523bb3c83365eaf0ad4a2529ecf"}
Sep 07 21:16:17.042577 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:17.042177    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:17.043638 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:17.043609    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt" event={"ID":"9d42deef-a708-4caf-bcd2-7b170e361a35","Type":"ContainerStarted","Data":"51ab83e642731e92924fa6c3e8b25b7601f001bd73ff2602ef7b9d3aa70e1f94"}
Sep 07 21:16:17.043785 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:17.043657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:17.058723 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:17.058676    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf" podStartSLOduration=1.724416652 podStartE2EDuration="4.05866249s" totalImagesPullingTime="2.334245838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:16:17.056919738 +0000 UTC m=+869.138447081" watchObservedRunningTime="2026-09-07 21:16:17.05866249 +0000 UTC m=+869.140189844"
Sep 07 21:16:17.073595 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:17.073550    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt" podStartSLOduration=1.7228439020000001 podStartE2EDuration="4.073537556s" totalImagesPullingTime="2.350693654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:16:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:16:17.071550984 +0000 UTC m=+869.153078326" watchObservedRunningTime="2026-09-07 21:16:17.073537556 +0000 UTC m=+869.155064897"
Sep 07 21:16:17.262234 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:17.262191    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:16:24.235755 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.235716    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"]
Sep 07 21:16:24.239227 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.239205    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.241374 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.241352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-9e7d7ca5-kserve-self-signed-certs\""
Sep 07 21:16:24.250996 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.250957    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"]
Sep 07 21:16:24.343406 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.343373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tls-certs\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.343406 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.343412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-dshm\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.343613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.343434    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tmp-dir\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.343613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.343499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzp7t\" (UniqueName: \"kubernetes.io/projected/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-kube-api-access-hzp7t\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.343613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.343549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-home\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.343613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.343574    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-model-cache\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.444582 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.444549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tls-certs\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.444582 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.444594    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-dshm\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.444843 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.444623    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tmp-dir\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.444843 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.444656    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzp7t\" (UniqueName: \"kubernetes.io/projected/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-kube-api-access-hzp7t\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.444843 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.444687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-home\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.444843 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.444707    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-model-cache\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.445074 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.445050    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tmp-dir\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.445518 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.445489    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-model-cache\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.445643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.445523    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-home\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.446992 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.446945    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-dshm\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.447524 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.447500    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tls-certs\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.452401 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.452376    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzp7t\" (UniqueName: \"kubernetes.io/projected/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-kube-api-access-hzp7t\") pod \"e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") " pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.551050 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.550982    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:24.826216 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:24.826184    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"]
Sep 07 21:16:24.827807 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:16:24.827775    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd1c5c8fd_e25f_4c2d_871d_baeaf7841849.slice/crio-0574f1d4f600715cf1bfcf83c5508af00afc0b3239e406f49b0900e70dbb1529 WatchSource:0}: Error finding container 0574f1d4f600715cf1bfcf83c5508af00afc0b3239e406f49b0900e70dbb1529: Status 404 returned error can't find the container with id 0574f1d4f600715cf1bfcf83c5508af00afc0b3239e406f49b0900e70dbb1529
Sep 07 21:16:25.100268 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:25.100229    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8" event={"ID":"d1c5c8fd-e25f-4c2d-871d-baeaf7841849","Type":"ContainerStarted","Data":"51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b"}
Sep 07 21:16:25.100447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:25.100285    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8" event={"ID":"d1c5c8fd-e25f-4c2d-871d-baeaf7841849","Type":"ContainerStarted","Data":"0574f1d4f600715cf1bfcf83c5508af00afc0b3239e406f49b0900e70dbb1529"}
Sep 07 21:16:25.116247 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:25.116196    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8" podStartSLOduration=1.116174674 podStartE2EDuration="1.116174674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:16:25.115005459 +0000 UTC m=+877.196532798" watchObservedRunningTime="2026-09-07 21:16:25.116174674 +0000 UTC m=+877.197702014"
Sep 07 21:16:27.039981 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:27.038910    2565 patch_prober.go:28] interesting pod/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:38742->10.134.0.37:15021: read: connection reset by peer" start-of-body=
Sep 07 21:16:27.039981 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:27.039034    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": read tcp 10.134.0.2:38742->10.134.0.37:15021: read: connection reset by peer"
Sep 07 21:16:27.114506 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:27.114479    2565 generic.go:356] "Generic (PLEG): container finished" podID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerID="4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb" exitCode=0
Sep 07 21:16:27.114627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:27.114560    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" event={"ID":"b6365827-6045-41c2-a858-a2b06a8cf7eb","Type":"ContainerDied","Data":"4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb"}
Sep 07 21:16:27.114971 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:27.114946    2565 scope.go:117] "RemoveContainer" containerID="4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb"
Sep 07 21:16:27.116039 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:27.116013    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:27.116097 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:27.116086    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:27.116154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:27.116142    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:28.056383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:28.056351    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:28.057020 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:28.056995    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:28.122152 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:28.122117    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" event={"ID":"b6365827-6045-41c2-a858-a2b06a8cf7eb","Type":"ContainerStarted","Data":"dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62"}
Sep 07 21:16:28.207387 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:28.207354    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:28.207861 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:28.207830    2565 patch_prober.go:28] interesting pod/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 21:16:28.208020 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:28.207897    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 21:16:29.208049 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:29.208014    2565 patch_prober.go:28] interesting pod/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 21:16:29.208431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:29.208063    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 21:16:30.207692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:30.207653    2565 patch_prober.go:28] interesting pod/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 21:16:30.207874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:30.207714    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 21:16:31.207883 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:31.207850    2565 patch_prober.go:28] interesting pod/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 21:16:31.208297 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:31.207905    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 21:16:32.208420 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:32.208381    2565 patch_prober.go:28] interesting pod/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 21:16:32.208815 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:32.208435    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 21:16:33.208243 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:33.208198    2565 patch_prober.go:28] interesting pod/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 21:16:33.208426 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:33.208260    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 21:16:34.211150 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:34.211117    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:34.211531 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:34.211403    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:34.212120 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:34.212098    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:34.551740 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:34.551658    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:34.551740 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:34.551719    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:34.564852 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:34.564822    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:35.166227 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:35.166198    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:16:36.252465 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:36.252432    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:16:37.389715 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.389679    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"]
Sep 07 21:16:37.393181 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.393165    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"
Sep 07 21:16:37.395415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.395378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:16:37.398715 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.395788    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:16:37.401077 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.401048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"]
Sep 07 21:16:37.470781 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.470754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b11cfe-dbb9-40b4-a944-ae1e55f580f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r\" (UID: \"92b11cfe-dbb9-40b4-a944-ae1e55f580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"
Sep 07 21:16:37.571425 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.571392    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b11cfe-dbb9-40b4-a944-ae1e55f580f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r\" (UID: \"92b11cfe-dbb9-40b4-a944-ae1e55f580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"
Sep 07 21:16:37.572116 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.572098    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b11cfe-dbb9-40b4-a944-ae1e55f580f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r\" (UID: \"92b11cfe-dbb9-40b4-a944-ae1e55f580f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"
Sep 07 21:16:37.725205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.725126    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"
Sep 07 21:16:37.992018 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:37.991990    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"]
Sep 07 21:16:37.994344 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:16:37.994314    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b11cfe_dbb9_40b4_a944_ae1e55f580f0.slice/crio-517ef7b568b42592d1e724cc848ffaf45bc8fd593e6086b4b036b21a72d06015 WatchSource:0}: Error finding container 517ef7b568b42592d1e724cc848ffaf45bc8fd593e6086b4b036b21a72d06015: Status 404 returned error can't find the container with id 517ef7b568b42592d1e724cc848ffaf45bc8fd593e6086b4b036b21a72d06015
Sep 07 21:16:38.169762 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:38.169725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r" event={"ID":"92b11cfe-dbb9-40b4-a944-ae1e55f580f0","Type":"ContainerStarted","Data":"517ef7b568b42592d1e724cc848ffaf45bc8fd593e6086b4b036b21a72d06015"}
Sep 07 21:16:39.176198 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:39.176162    2565 generic.go:356] "Generic (PLEG): container finished" podID="92b11cfe-dbb9-40b4-a944-ae1e55f580f0" containerID="487aac6e20b7eac46a4b6a89e926a6025b22857decabbc56599f41a67506dfd8" exitCode=0
Sep 07 21:16:39.176562 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:39.176233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r" event={"ID":"92b11cfe-dbb9-40b4-a944-ae1e55f580f0","Type":"ContainerDied","Data":"487aac6e20b7eac46a4b6a89e926a6025b22857decabbc56599f41a67506dfd8"}
Sep 07 21:16:40.312570 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:40.312539    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"
Sep 07 21:16:40.394785 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:40.394755    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b11cfe-dbb9-40b4-a944-ae1e55f580f0-maas-api-service-ca\") pod \"92b11cfe-dbb9-40b4-a944-ae1e55f580f0\" (UID: \"92b11cfe-dbb9-40b4-a944-ae1e55f580f0\") "
Sep 07 21:16:40.395103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:40.395080    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b11cfe-dbb9-40b4-a944-ae1e55f580f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92b11cfe-dbb9-40b4-a944-ae1e55f580f0" (UID: "92b11cfe-dbb9-40b4-a944-ae1e55f580f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:16:40.495875 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:40.495803    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92b11cfe-dbb9-40b4-a944-ae1e55f580f0-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:41.186553 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:41.186516    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r" event={"ID":"92b11cfe-dbb9-40b4-a944-ae1e55f580f0","Type":"ContainerDied","Data":"517ef7b568b42592d1e724cc848ffaf45bc8fd593e6086b4b036b21a72d06015"}
Sep 07 21:16:41.186722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:41.186566    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="517ef7b568b42592d1e724cc848ffaf45bc8fd593e6086b4b036b21a72d06015"
Sep 07 21:16:41.186722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:41.186532    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"
Sep 07 21:16:44.005616 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.005583    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"]
Sep 07 21:16:44.006025 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.005825    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf" podUID="9c294631-e3ae-4689-9a3f-4a3b7cc8ab66" containerName="payload-processing" containerID="cri-o://6e3aff369caeb7a91b44f3f476fa4a6877f9e523bb3c83365eaf0ad4a2529ecf" gracePeriod=30
Sep 07 21:16:44.011193 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.011164    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"]
Sep 07 21:16:44.011404 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.011360    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt" podUID="9d42deef-a708-4caf-bcd2-7b170e361a35" containerName="payload-pre-processing" containerID="cri-o://51ab83e642731e92924fa6c3e8b25b7601f001bd73ff2602ef7b9d3aa70e1f94" gracePeriod=30
Sep 07 21:16:44.161956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.161889    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"]
Sep 07 21:16:44.164200 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.164174    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-f86b6ac3-6vz2r"]
Sep 07 21:16:44.203163 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.203136    2565 generic.go:356] "Generic (PLEG): container finished" podID="9c294631-e3ae-4689-9a3f-4a3b7cc8ab66" containerID="6e3aff369caeb7a91b44f3f476fa4a6877f9e523bb3c83365eaf0ad4a2529ecf" exitCode=0
Sep 07 21:16:44.203298 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.203210    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf" event={"ID":"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66","Type":"ContainerDied","Data":"6e3aff369caeb7a91b44f3f476fa4a6877f9e523bb3c83365eaf0ad4a2529ecf"}
Sep 07 21:16:44.204828 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.204808    2565 generic.go:356] "Generic (PLEG): container finished" podID="9d42deef-a708-4caf-bcd2-7b170e361a35" containerID="51ab83e642731e92924fa6c3e8b25b7601f001bd73ff2602ef7b9d3aa70e1f94" exitCode=0
Sep 07 21:16:44.204938 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.204845    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt" event={"ID":"9d42deef-a708-4caf-bcd2-7b170e361a35","Type":"ContainerDied","Data":"51ab83e642731e92924fa6c3e8b25b7601f001bd73ff2602ef7b9d3aa70e1f94"}
Sep 07 21:16:44.291639 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.291530    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"]
Sep 07 21:16:44.291918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.291774    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" containerID="cri-o://dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62" gracePeriod=30
Sep 07 21:16:44.302225 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.302152    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:44.306503 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.306482    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:44.364848 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.364817    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92b11cfe-dbb9-40b4-a944-ae1e55f580f0" path="/var/lib/kubelet/pods/92b11cfe-dbb9-40b4-a944-ae1e55f580f0/volumes"
Sep 07 21:16:44.434543 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.434505    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d42deef-a708-4caf-bcd2-7b170e361a35-plugins-config-volume\") pod \"9d42deef-a708-4caf-bcd2-7b170e361a35\" (UID: \"9d42deef-a708-4caf-bcd2-7b170e361a35\") "
Sep 07 21:16:44.434730 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.434587    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-plugins-config-volume\") pod \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\" (UID: \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\") "
Sep 07 21:16:44.434730 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.434624    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sdbv\" (UniqueName: \"kubernetes.io/projected/9d42deef-a708-4caf-bcd2-7b170e361a35-kube-api-access-6sdbv\") pod \"9d42deef-a708-4caf-bcd2-7b170e361a35\" (UID: \"9d42deef-a708-4caf-bcd2-7b170e361a35\") "
Sep 07 21:16:44.434730 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.434645    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rp2cn\" (UniqueName: \"kubernetes.io/projected/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-kube-api-access-rp2cn\") pod \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\" (UID: \"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66\") "
Sep 07 21:16:44.435010 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.434953    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d42deef-a708-4caf-bcd2-7b170e361a35-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9d42deef-a708-4caf-bcd2-7b170e361a35" (UID: "9d42deef-a708-4caf-bcd2-7b170e361a35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:16:44.435103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.434986    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c294631-e3ae-4689-9a3f-4a3b7cc8ab66" (UID: "9c294631-e3ae-4689-9a3f-4a3b7cc8ab66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:16:44.436727 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.436703    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-kube-api-access-rp2cn" (OuterVolumeSpecName: "kube-api-access-rp2cn") pod "9c294631-e3ae-4689-9a3f-4a3b7cc8ab66" (UID: "9c294631-e3ae-4689-9a3f-4a3b7cc8ab66"). InnerVolumeSpecName "kube-api-access-rp2cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:16:44.436815 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.436789    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d42deef-a708-4caf-bcd2-7b170e361a35-kube-api-access-6sdbv" (OuterVolumeSpecName: "kube-api-access-6sdbv") pod "9d42deef-a708-4caf-bcd2-7b170e361a35" (UID: "9d42deef-a708-4caf-bcd2-7b170e361a35"). InnerVolumeSpecName "kube-api-access-6sdbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:16:44.536238 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.536141    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d42deef-a708-4caf-bcd2-7b170e361a35-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:44.536238 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.536184    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:44.536238 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.536199    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sdbv\" (UniqueName: \"kubernetes.io/projected/9d42deef-a708-4caf-bcd2-7b170e361a35-kube-api-access-6sdbv\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:44.536238 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.536214    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp2cn\" (UniqueName: \"kubernetes.io/projected/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66-kube-api-access-rp2cn\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:44.983766 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.983733    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"]
Sep 07 21:16:44.984141 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984120    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c294631-e3ae-4689-9a3f-4a3b7cc8ab66" containerName="payload-processing"
Sep 07 21:16:44.984141 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984139    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c294631-e3ae-4689-9a3f-4a3b7cc8ab66" containerName="payload-processing"
Sep 07 21:16:44.984302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984153    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d42deef-a708-4caf-bcd2-7b170e361a35" containerName="payload-pre-processing"
Sep 07 21:16:44.984302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984159    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d42deef-a708-4caf-bcd2-7b170e361a35" containerName="payload-pre-processing"
Sep 07 21:16:44.984302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984174    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92b11cfe-dbb9-40b4-a944-ae1e55f580f0" containerName="revoke-keys"
Sep 07 21:16:44.984302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984182    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="92b11cfe-dbb9-40b4-a944-ae1e55f580f0" containerName="revoke-keys"
Sep 07 21:16:44.984302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984273    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c294631-e3ae-4689-9a3f-4a3b7cc8ab66" containerName="payload-processing"
Sep 07 21:16:44.984302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984285    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="92b11cfe-dbb9-40b4-a944-ae1e55f580f0" containerName="revoke-keys"
Sep 07 21:16:44.984302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.984298    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d42deef-a708-4caf-bcd2-7b170e361a35" containerName="payload-pre-processing"
Sep 07 21:16:44.987190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.987172    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:44.989687 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:44.989670    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b910426f-openshift-default-dockercfg-cz6pv\""
Sep 07 21:16:45.002953 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.002928    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"]
Sep 07 21:16:45.142360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142306    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-credential-socket\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.142792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-socket\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.142792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-data\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.142792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-envoy\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.142792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142549    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zspd\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-kube-api-access-2zspd\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.142792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142609    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-token\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.142792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142637    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istiod-ca-cert\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.142792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-certs\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.142792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.142723    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-podinfo\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.211606 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.211575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"
Sep 07 21:16:45.211792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.211576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf" event={"ID":"9c294631-e3ae-4689-9a3f-4a3b7cc8ab66","Type":"ContainerDied","Data":"353b93ea3df2ab943af4e1adef291ce8e477fb022c5c34f18bb8f8c16318cf5d"}
Sep 07 21:16:45.211792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.211718    2565 scope.go:117] "RemoveContainer" containerID="6e3aff369caeb7a91b44f3f476fa4a6877f9e523bb3c83365eaf0ad4a2529ecf"
Sep 07 21:16:45.213323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.213195    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt" event={"ID":"9d42deef-a708-4caf-bcd2-7b170e361a35","Type":"ContainerDied","Data":"a33cf7eeb4e43022a6d05ee1ec65a4b25a864289e036a428c7ea22fda381914a"}
Sep 07 21:16:45.213323 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.213232    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"
Sep 07 21:16:45.223440 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.223416    2565 scope.go:117] "RemoveContainer" containerID="51ab83e642731e92924fa6c3e8b25b7601f001bd73ff2602ef7b9d3aa70e1f94"
Sep 07 21:16:45.238944 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.238888    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"]
Sep 07 21:16:45.242433 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.242412    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f86b6ac3-8dbb8cb46-q5hcf"]
Sep 07 21:16:45.243330 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zspd\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-kube-api-access-2zspd\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.243436 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-token\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.243436 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243365    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istiod-ca-cert\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.243575 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-certs\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.243630 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243607    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-podinfo\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.243687 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-credential-socket\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.243737 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243717    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-socket\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.243778 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243768    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-data\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.243823 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243799    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-envoy\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.244173 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243950    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-certs\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.244173 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.243996    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-credential-socket\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.244173 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.244118    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-socket\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.244341 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.244257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istiod-ca-cert\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.244341 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.244265    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-data\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.246179 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.246158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-podinfo\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.246359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.246344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-envoy\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.253538 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.253514    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"]
Sep 07 21:16:45.255643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.255612    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-token\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.255754 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.255615    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zspd\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-kube-api-access-2zspd\") pod \"e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") " pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.257135 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.257112    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f86b6ac3-778b98777d-vzrxt"]
Sep 07 21:16:45.298228 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.298195    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:45.579153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.579060    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"]
Sep 07 21:16:45.581876 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:16:45.581852    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf565515_25ee_4fd9_91f3_2c499e1bb2e6.slice/crio-7071b3f25157cdf3c5dd1585f6a4ea52a19a33c36837d5ba1114ad4822ca4205 WatchSource:0}: Error finding container 7071b3f25157cdf3c5dd1585f6a4ea52a19a33c36837d5ba1114ad4822ca4205: Status 404 returned error can't find the container with id 7071b3f25157cdf3c5dd1585f6a4ea52a19a33c36837d5ba1114ad4822ca4205
Sep 07 21:16:45.584214 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.584182    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:45.584296 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.584246    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:45.584296 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:45.584274    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:16:46.220860 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:46.220821    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" event={"ID":"af565515-25ee-4fd9-91f3-2c499e1bb2e6","Type":"ContainerStarted","Data":"ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6"}
Sep 07 21:16:46.221284 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:46.220877    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" event={"ID":"af565515-25ee-4fd9-91f3-2c499e1bb2e6","Type":"ContainerStarted","Data":"7071b3f25157cdf3c5dd1585f6a4ea52a19a33c36837d5ba1114ad4822ca4205"}
Sep 07 21:16:46.241145 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:46.241097    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" podStartSLOduration=2.24108213 podStartE2EDuration="2.24108213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:16:46.239532045 +0000 UTC m=+898.321059387" watchObservedRunningTime="2026-09-07 21:16:46.24108213 +0000 UTC m=+898.322609471"
Sep 07 21:16:46.298404 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:46.298374    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:46.304382 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:46.304356    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:46.364174 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:46.364139    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c294631-e3ae-4689-9a3f-4a3b7cc8ab66" path="/var/lib/kubelet/pods/9c294631-e3ae-4689-9a3f-4a3b7cc8ab66/volumes"
Sep 07 21:16:46.364497 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:46.364484    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d42deef-a708-4caf-bcd2-7b170e361a35" path="/var/lib/kubelet/pods/9d42deef-a708-4caf-bcd2-7b170e361a35/volumes"
Sep 07 21:16:47.226434 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:47.226404    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:47.227447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:47.227424    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:16:49.214028 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.213994    2565 patch_prober.go:28] interesting pod/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 21:16:49.214400 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.214053    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 21:16:49.323937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.323910    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"]
Sep 07 21:16:49.327286 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.327253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:49.329856 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.329834    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b910426f-dockercfg-km6zf\""
Sep 07 21:16:49.329856 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.329850    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b910426f\""
Sep 07 21:16:49.337671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.337652    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"]
Sep 07 21:16:49.486692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.486603    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-plugins-config-volume\") pod \"payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz\" (UID: \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\") " pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:49.486850 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.486703    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxrpm\" (UniqueName: \"kubernetes.io/projected/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-kube-api-access-kxrpm\") pod \"payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz\" (UID: \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\") " pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:49.546588 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.546564    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:49.588025 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.587998    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-plugins-config-volume\") pod \"payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz\" (UID: \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\") " pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:49.588179 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.588087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxrpm\" (UniqueName: \"kubernetes.io/projected/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-kube-api-access-kxrpm\") pod \"payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz\" (UID: \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\") " pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:49.588732 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.588709    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-plugins-config-volume\") pod \"payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz\" (UID: \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\") " pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:49.596246 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.596215    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxrpm\" (UniqueName: \"kubernetes.io/projected/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-kube-api-access-kxrpm\") pod \"payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz\" (UID: \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\") " pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:49.678321 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.678287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:49.688902 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.688880    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6365827-6045-41c2-a858-a2b06a8cf7eb-istiod-ca-cert\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689034 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.688918    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-certs\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689034 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.688978    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-credential-socket\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689034 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689018    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-envoy\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689192 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689060    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-data\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689192 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689124    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8qr2\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-kube-api-access-c8qr2\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689198    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-podinfo\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689240    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:16:49.689413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689252    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-token\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689253    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:16:49.689413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689280    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-socket\") pod \"b6365827-6045-41c2-a858-a2b06a8cf7eb\" (UID: \"b6365827-6045-41c2-a858-a2b06a8cf7eb\") "
Sep 07 21:16:49.689413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689294    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6365827-6045-41c2-a858-a2b06a8cf7eb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:16:49.689708 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689505    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:16:49.689708 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689592    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6365827-6045-41c2-a858-a2b06a8cf7eb-istiod-ca-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.689708 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689611    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.689708 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689624    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-credential-socket\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.689708 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689637    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-data\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.689992 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.689729    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:16:49.691496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.691466    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:16:49.691643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.691619    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 21:16:49.691717 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.691696    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-kube-api-access-c8qr2" (OuterVolumeSpecName: "kube-api-access-c8qr2") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "kube-api-access-c8qr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:16:49.691992 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.691954    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6365827-6045-41c2-a858-a2b06a8cf7eb" (UID: "b6365827-6045-41c2-a858-a2b06a8cf7eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:16:49.790801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.790723    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8qr2\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-kube-api-access-c8qr2\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.790801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.790753    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-podinfo\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.790801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.790762    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-token\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.790801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.790772    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-workload-socket\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.790801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.790779    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6365827-6045-41c2-a858-a2b06a8cf7eb-istio-envoy\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:16:49.948020 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:49.947986    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"]
Sep 07 21:16:49.950058 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:16:49.950021    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8beccc6d_c7a5_4e63_b2c6_bedc03df605a.slice/crio-2893770c17669ab150455819a037dd23663748e8af55f8baf565ee977e71c18c WatchSource:0}: Error finding container 2893770c17669ab150455819a037dd23663748e8af55f8baf565ee977e71c18c: Status 404 returned error can't find the container with id 2893770c17669ab150455819a037dd23663748e8af55f8baf565ee977e71c18c
Sep 07 21:16:50.241857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.241802    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz" event={"ID":"8beccc6d-c7a5-4e63-b2c6-bedc03df605a","Type":"ContainerStarted","Data":"c4318e779d611ee9f8803545202c8219f2c143592264701fbff370fcb1b7bdfe"}
Sep 07 21:16:50.242304 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.242008    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:16:50.242304 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.242065    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz" event={"ID":"8beccc6d-c7a5-4e63-b2c6-bedc03df605a","Type":"ContainerStarted","Data":"2893770c17669ab150455819a037dd23663748e8af55f8baf565ee977e71c18c"}
Sep 07 21:16:50.243589 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.243560    2565 generic.go:356] "Generic (PLEG): container finished" podID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerID="dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62" exitCode=0
Sep 07 21:16:50.243708 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.243615    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"
Sep 07 21:16:50.243708 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.243619    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" event={"ID":"b6365827-6045-41c2-a858-a2b06a8cf7eb","Type":"ContainerDied","Data":"dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62"}
Sep 07 21:16:50.244112 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.243744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97" event={"ID":"b6365827-6045-41c2-a858-a2b06a8cf7eb","Type":"ContainerDied","Data":"d0f1979c3c286dceac5dcf3777e54a9afc906a34d309d6e91f7db6f0be44aa57"}
Sep 07 21:16:50.244112 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.243781    2565 scope.go:117] "RemoveContainer" containerID="dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62"
Sep 07 21:16:50.253569 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.253555    2565 scope.go:117] "RemoveContainer" containerID="4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb"
Sep 07 21:16:50.259576 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.259536    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz" podStartSLOduration=1.259523587 podStartE2EDuration="1.259523587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:16:50.257468306 +0000 UTC m=+902.338995648" watchObservedRunningTime="2026-09-07 21:16:50.259523587 +0000 UTC m=+902.341050928"
Sep 07 21:16:50.262170 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.262154    2565 scope.go:117] "RemoveContainer" containerID="dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62"
Sep 07 21:16:50.262454 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:16:50.262436    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62\": container with ID starting with dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62 not found: ID does not exist" containerID="dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62"
Sep 07 21:16:50.262714 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.262463    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62"} err="failed to get container status \"dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62\": rpc error: code = NotFound desc = could not find container \"dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62\": container with ID starting with dbef2d268c5bb20217b088b4a7774d23d9ef64929c808f5fc524064d24b41c62 not found: ID does not exist"
Sep 07 21:16:50.262714 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.262480    2565 scope.go:117] "RemoveContainer" containerID="4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb"
Sep 07 21:16:50.262882 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:16:50.262793    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb\": container with ID starting with 4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb not found: ID does not exist" containerID="4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb"
Sep 07 21:16:50.262882 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.262807    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb"} err="failed to get container status \"4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb\": rpc error: code = NotFound desc = could not find container \"4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb\": container with ID starting with 4e124a9eb4ee507f1cb5c5418b6695f7ebd1307b1b2e704db2e8d31ac9627bcb not found: ID does not exist"
Sep 07 21:16:50.271372 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.271351    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"]
Sep 07 21:16:50.275413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.275393    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f86b6ac3-gw-openshift-default-548d6cc656-zlh97"]
Sep 07 21:16:50.364975 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:50.364938    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" path="/var/lib/kubelet/pods/b6365827-6045-41c2-a858-a2b06a8cf7eb/volumes"
Sep 07 21:16:55.458825 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:16:55.458774    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:17:01.252150 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:01.252115    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:17:06.329488 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:06.329447    2565 generic.go:356] "Generic (PLEG): container finished" podID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerID="ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6" exitCode=0
Sep 07 21:17:06.329958 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:06.329521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" event={"ID":"af565515-25ee-4fd9-91f3-2c499e1bb2e6","Type":"ContainerDied","Data":"ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6"}
Sep 07 21:17:06.329958 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:06.329915    2565 scope.go:117] "RemoveContainer" containerID="ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6"
Sep 07 21:17:06.331035 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:06.331009    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:17:06.331100 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:06.331083    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:17:06.331138 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:06.331117    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:17:07.336214 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:07.336182    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" event={"ID":"af565515-25ee-4fd9-91f3-2c499e1bb2e6","Type":"ContainerStarted","Data":"7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2"}
Sep 07 21:17:08.298499 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:08.298460    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:17:08.298700 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:08.298678    2565 patch_prober.go:28] interesting pod/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 07 21:17:08.298747 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:08.298727    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 07 21:17:09.299495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:09.299452    2565 patch_prober.go:28] interesting pod/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 07 21:17:09.300009 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:09.299528    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 07 21:17:10.299003 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:10.298938    2565 patch_prober.go:28] interesting pod/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 07 21:17:10.299192 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:10.299021    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 07 21:17:11.299360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.299321    2565 patch_prober.go:28] interesting pod/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 07 21:17:11.299755 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.299381    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 07 21:17:11.527733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.527702    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"]
Sep 07 21:17:11.528073 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.528060    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy"
Sep 07 21:17:11.528123 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.528074    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy"
Sep 07 21:17:11.528123 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.528092    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy"
Sep 07 21:17:11.528123 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.528097    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy"
Sep 07 21:17:11.528220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.528149    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy"
Sep 07 21:17:11.528220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.528158    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6365827-6045-41c2-a858-a2b06a8cf7eb" containerName="istio-proxy"
Sep 07 21:17:11.531281 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.531265    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"
Sep 07 21:17:11.533596 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.533572    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:17:11.533742 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.533574    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:17:11.538612 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.538587    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"]
Sep 07 21:17:11.591428 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.591402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42301cd-2f8e-4b53-b1f9-5cb79690aa86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs\" (UID: \"c42301cd-2f8e-4b53-b1f9-5cb79690aa86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"
Sep 07 21:17:11.692155 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.692114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42301cd-2f8e-4b53-b1f9-5cb79690aa86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs\" (UID: \"c42301cd-2f8e-4b53-b1f9-5cb79690aa86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"
Sep 07 21:17:11.692845 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.692817    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42301cd-2f8e-4b53-b1f9-5cb79690aa86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs\" (UID: \"c42301cd-2f8e-4b53-b1f9-5cb79690aa86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"
Sep 07 21:17:11.763621 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.763584    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"]
Sep 07 21:17:11.767564 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.767542    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.771466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.770175    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-06f0f6\"/\"kube-root-ca.crt\""
Sep 07 21:17:11.771466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.770573    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-06f0f6\"/\"openshift-service-ca.crt\""
Sep 07 21:17:11.771466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.770783    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-06f0f6\"/\"test-model-06f0f6-kserve-self-signed-certs\""
Sep 07 21:17:11.771466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.770996    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-06f0f6\"/\"default-dockercfg-lgqn5\""
Sep 07 21:17:11.775471 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.775159    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"]
Sep 07 21:17:11.793155 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.793127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tmp-dir\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.793264 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.793207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-dshm\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.793313 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.793273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-home\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.793352 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.793309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5cxx\" (UniqueName: \"kubernetes.io/projected/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-kube-api-access-b5cxx\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.793391 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.793352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-model-cache\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.793424 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.793394    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tls-certs\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.842766 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.842702    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"
Sep 07 21:17:11.894593 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.894566    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-dshm\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.894747 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.894618    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-home\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.894747 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.894728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5cxx\" (UniqueName: \"kubernetes.io/projected/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-kube-api-access-b5cxx\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.894828 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.894795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-model-cache\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.894863 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.894845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tls-certs\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.894912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.894895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tmp-dir\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.895043 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.895024    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-home\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.895242 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.895219    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tmp-dir\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.895320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.895304    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-model-cache\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.896812 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.896784    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-dshm\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.897609 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.897587    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tls-certs\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:11.905182 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:11.905158    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5cxx\" (UniqueName: \"kubernetes.io/projected/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-kube-api-access-b5cxx\") pod \"test-model-06f0f6-kserve-76c4cd6997-4vv7x\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") " pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:12.083232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:12.083192    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:12.141244 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:12.141164    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"]
Sep 07 21:17:12.298783 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:12.298752    2565 patch_prober.go:28] interesting pod/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 07 21:17:12.298946 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:12.298804    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 07 21:17:12.371010 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:12.367150    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"]
Sep 07 21:17:12.371010 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:12.369004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs" event={"ID":"c42301cd-2f8e-4b53-b1f9-5cb79690aa86","Type":"ContainerStarted","Data":"c4ee6ea4fbba59339cde4997d3328d6a1b8307ef5809befa1aa6209cedcfb34e"}
Sep 07 21:17:12.951764 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:12.951730    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:17:13.258893 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.258806    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"]
Sep 07 21:17:13.262061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.262044    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"
Sep 07 21:17:13.268547 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.268517    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"]
Sep 07 21:17:13.302422 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.302391    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:17:13.302672 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.302652    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:17:13.303463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.303444    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:17:13.306544 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.306524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bfb8b1-bc0d-4876-a57e-814889ee5999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b910426f-42xdz\" (UID: \"f8bfb8b1-bc0d-4876-a57e-814889ee5999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"
Sep 07 21:17:13.376983 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.376939    2565 generic.go:356] "Generic (PLEG): container finished" podID="c42301cd-2f8e-4b53-b1f9-5cb79690aa86" containerID="386b892a1a20c768ca61cc452791a955d7bd05a04aad99c5506b300bee00ca17" exitCode=0
Sep 07 21:17:13.377402 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.377005    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs" event={"ID":"c42301cd-2f8e-4b53-b1f9-5cb79690aa86","Type":"ContainerDied","Data":"386b892a1a20c768ca61cc452791a955d7bd05a04aad99c5506b300bee00ca17"}
Sep 07 21:17:13.378718 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.378689    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x" event={"ID":"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc","Type":"ContainerStarted","Data":"322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430"}
Sep 07 21:17:13.378868 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.378725    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x" event={"ID":"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc","Type":"ContainerStarted","Data":"478f3ce65ec9e828ac3af0426cb348f76f3c2eedfb88a5891b73079f75e84774"}
Sep 07 21:17:13.407492 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.407458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bfb8b1-bc0d-4876-a57e-814889ee5999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b910426f-42xdz\" (UID: \"f8bfb8b1-bc0d-4876-a57e-814889ee5999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"
Sep 07 21:17:13.407635 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.407552    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x" podStartSLOduration=2.407537858 podStartE2EDuration="2.407537858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:17:13.406625454 +0000 UTC m=+925.488152806" watchObservedRunningTime="2026-09-07 21:17:13.407537858 +0000 UTC m=+925.489065199"
Sep 07 21:17:13.408169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.408149    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bfb8b1-bc0d-4876-a57e-814889ee5999-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b910426f-42xdz\" (UID: \"f8bfb8b1-bc0d-4876-a57e-814889ee5999\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"
Sep 07 21:17:13.574329 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.574253    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"
Sep 07 21:17:13.772617 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.772577    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"]
Sep 07 21:17:13.843077 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:13.843053    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"]
Sep 07 21:17:13.845288 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:17:13.845260    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8bfb8b1_bc0d_4876_a57e_814889ee5999.slice/crio-77ea173484984b456b9173ceb733a0faba34ae4cbbe41ce99f94b7c60c36bd4a WatchSource:0}: Error finding container 77ea173484984b456b9173ceb733a0faba34ae4cbbe41ce99f94b7c60c36bd4a: Status 404 returned error can't find the container with id 77ea173484984b456b9173ceb733a0faba34ae4cbbe41ce99f94b7c60c36bd4a
Sep 07 21:17:14.384138 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:14.384102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz" event={"ID":"f8bfb8b1-bc0d-4876-a57e-814889ee5999","Type":"ContainerStarted","Data":"44b85ecaa3a3b884da00a479fe6bf10cb84e25bcaa65a2db563d353c5eaeb006"}
Sep 07 21:17:14.384494 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:14.384165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz" event={"ID":"f8bfb8b1-bc0d-4876-a57e-814889ee5999","Type":"ContainerStarted","Data":"77ea173484984b456b9173ceb733a0faba34ae4cbbe41ce99f94b7c60c36bd4a"}
Sep 07 21:17:14.404287 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:14.402181    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz" podStartSLOduration=1.402153546 podStartE2EDuration="1.402153546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:17:14.39853812 +0000 UTC m=+926.480065465" watchObservedRunningTime="2026-09-07 21:17:14.402153546 +0000 UTC m=+926.483680887"
Sep 07 21:17:14.548405 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:14.548380    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"
Sep 07 21:17:14.618388 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:14.618358    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42301cd-2f8e-4b53-b1f9-5cb79690aa86-maas-api-service-ca\") pod \"c42301cd-2f8e-4b53-b1f9-5cb79690aa86\" (UID: \"c42301cd-2f8e-4b53-b1f9-5cb79690aa86\") "
Sep 07 21:17:14.618660 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:14.618639    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c42301cd-2f8e-4b53-b1f9-5cb79690aa86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c42301cd-2f8e-4b53-b1f9-5cb79690aa86" (UID: "c42301cd-2f8e-4b53-b1f9-5cb79690aa86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:17:14.719809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:14.719742    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c42301cd-2f8e-4b53-b1f9-5cb79690aa86-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:15.391380 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:15.391357    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"
Sep 07 21:17:15.391780 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:15.391399    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs" event={"ID":"c42301cd-2f8e-4b53-b1f9-5cb79690aa86","Type":"ContainerDied","Data":"c4ee6ea4fbba59339cde4997d3328d6a1b8307ef5809befa1aa6209cedcfb34e"}
Sep 07 21:17:15.391780 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:15.391454    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4ee6ea4fbba59339cde4997d3328d6a1b8307ef5809befa1aa6209cedcfb34e"
Sep 07 21:17:15.393297 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:15.393273    2565 generic.go:356] "Generic (PLEG): container finished" podID="f8bfb8b1-bc0d-4876-a57e-814889ee5999" containerID="44b85ecaa3a3b884da00a479fe6bf10cb84e25bcaa65a2db563d353c5eaeb006" exitCode=0
Sep 07 21:17:15.393426 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:15.393354    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz" event={"ID":"f8bfb8b1-bc0d-4876-a57e-814889ee5999","Type":"ContainerDied","Data":"44b85ecaa3a3b884da00a479fe6bf10cb84e25bcaa65a2db563d353c5eaeb006"}
Sep 07 21:17:16.440549 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:16.440515    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"]
Sep 07 21:17:16.539956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:16.539932    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"
Sep 07 21:17:16.638751 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:16.638726    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bfb8b1-bc0d-4876-a57e-814889ee5999-maas-api-service-ca\") pod \"f8bfb8b1-bc0d-4876-a57e-814889ee5999\" (UID: \"f8bfb8b1-bc0d-4876-a57e-814889ee5999\") "
Sep 07 21:17:16.639103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:16.639079    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8bfb8b1-bc0d-4876-a57e-814889ee5999-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8bfb8b1-bc0d-4876-a57e-814889ee5999" (UID: "f8bfb8b1-bc0d-4876-a57e-814889ee5999"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:17:16.739876 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:16.739796    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8bfb8b1-bc0d-4876-a57e-814889ee5999-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:17.403557 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:17.403519    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"
Sep 07 21:17:17.403719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:17.403552    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz" event={"ID":"f8bfb8b1-bc0d-4876-a57e-814889ee5999","Type":"ContainerDied","Data":"77ea173484984b456b9173ceb733a0faba34ae4cbbe41ce99f94b7c60c36bd4a"}
Sep 07 21:17:17.403719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:17.403603    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77ea173484984b456b9173ceb733a0faba34ae4cbbe41ce99f94b7c60c36bd4a"
Sep 07 21:17:19.301455 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.301422    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"]
Sep 07 21:17:19.301885 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.301707    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz" podUID="8beccc6d-c7a5-4e63-b2c6-bedc03df605a" containerName="payload-processing" containerID="cri-o://c4318e779d611ee9f8803545202c8219f2c143592264701fbff370fcb1b7bdfe" gracePeriod=30
Sep 07 21:17:19.419887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.419859    2565 generic.go:356] "Generic (PLEG): container finished" podID="8beccc6d-c7a5-4e63-b2c6-bedc03df605a" containerID="c4318e779d611ee9f8803545202c8219f2c143592264701fbff370fcb1b7bdfe" exitCode=0
Sep 07 21:17:19.420008 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.419926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz" event={"ID":"8beccc6d-c7a5-4e63-b2c6-bedc03df605a","Type":"ContainerDied","Data":"c4318e779d611ee9f8803545202c8219f2c143592264701fbff370fcb1b7bdfe"}
Sep 07 21:17:19.456820 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.456791    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"]
Sep 07 21:17:19.461382 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.461357    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b910426f-42xdz"]
Sep 07 21:17:19.568926 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.568870    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:17:19.668662 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.668629    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxrpm\" (UniqueName: \"kubernetes.io/projected/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-kube-api-access-kxrpm\") pod \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\" (UID: \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\") "
Sep 07 21:17:19.668662 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.668667    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-plugins-config-volume\") pod \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\" (UID: \"8beccc6d-c7a5-4e63-b2c6-bedc03df605a\") "
Sep 07 21:17:19.669082 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.669057    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8beccc6d-c7a5-4e63-b2c6-bedc03df605a" (UID: "8beccc6d-c7a5-4e63-b2c6-bedc03df605a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:17:19.670743 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.670723    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-kube-api-access-kxrpm" (OuterVolumeSpecName: "kube-api-access-kxrpm") pod "8beccc6d-c7a5-4e63-b2c6-bedc03df605a" (UID: "8beccc6d-c7a5-4e63-b2c6-bedc03df605a"). InnerVolumeSpecName "kube-api-access-kxrpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:17:19.769260 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.769222    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxrpm\" (UniqueName: \"kubernetes.io/projected/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-kube-api-access-kxrpm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:19.769260 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:19.769253    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8beccc6d-c7a5-4e63-b2c6-bedc03df605a-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:20.363537 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:20.363506    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8bfb8b1-bc0d-4876-a57e-814889ee5999" path="/var/lib/kubelet/pods/f8bfb8b1-bc0d-4876-a57e-814889ee5999/volumes"
Sep 07 21:17:20.426311 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:20.426283    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"
Sep 07 21:17:20.426453 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:20.426327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz" event={"ID":"8beccc6d-c7a5-4e63-b2c6-bedc03df605a","Type":"ContainerDied","Data":"2893770c17669ab150455819a037dd23663748e8af55f8baf565ee977e71c18c"}
Sep 07 21:17:20.426453 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:20.426386    2565 scope.go:117] "RemoveContainer" containerID="c4318e779d611ee9f8803545202c8219f2c143592264701fbff370fcb1b7bdfe"
Sep 07 21:17:20.445645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:20.445622    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"]
Sep 07 21:17:20.449362 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:20.449338    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b910426f-7765f84cff-mpgwz"]
Sep 07 21:17:20.519061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:20.518953    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"]
Sep 07 21:17:20.519240 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:20.519204    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy" containerID="cri-o://7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2" gracePeriod=30
Sep 07 21:17:22.084275 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:22.084240    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:22.084275 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:22.084286    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:22.097242 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:22.097214    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:22.365257 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:22.365184    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8beccc6d-c7a5-4e63-b2c6-bedc03df605a" path="/var/lib/kubelet/pods/8beccc6d-c7a5-4e63-b2c6-bedc03df605a/volumes"
Sep 07 21:17:22.448155 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:22.448128    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:17:22.561751 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:22.561698    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"]
Sep 07 21:17:22.563986 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:22.563951    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-9e1e8d-smdqs"]
Sep 07 21:17:24.370944 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:24.370899    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c42301cd-2f8e-4b53-b1f9-5cb79690aa86" path="/var/lib/kubelet/pods/c42301cd-2f8e-4b53-b1f9-5cb79690aa86/volumes"
Sep 07 21:17:25.509067 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509032    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"]
Sep 07 21:17:25.509444 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509387    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8beccc6d-c7a5-4e63-b2c6-bedc03df605a" containerName="payload-processing"
Sep 07 21:17:25.509444 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509400    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8beccc6d-c7a5-4e63-b2c6-bedc03df605a" containerName="payload-processing"
Sep 07 21:17:25.509444 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509421    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8bfb8b1-bc0d-4876-a57e-814889ee5999" containerName="revoke-keys"
Sep 07 21:17:25.509444 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509427    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8bfb8b1-bc0d-4876-a57e-814889ee5999" containerName="revoke-keys"
Sep 07 21:17:25.509444 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509446    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c42301cd-2f8e-4b53-b1f9-5cb79690aa86" containerName="revoke-keys"
Sep 07 21:17:25.509613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509453    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c42301cd-2f8e-4b53-b1f9-5cb79690aa86" containerName="revoke-keys"
Sep 07 21:17:25.509613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509516    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c42301cd-2f8e-4b53-b1f9-5cb79690aa86" containerName="revoke-keys"
Sep 07 21:17:25.509613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509525    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8bfb8b1-bc0d-4876-a57e-814889ee5999" containerName="revoke-keys"
Sep 07 21:17:25.509613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.509532    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8beccc6d-c7a5-4e63-b2c6-bedc03df605a" containerName="payload-processing"
Sep 07 21:17:25.513868 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.513844    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:25.516331 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.516301    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-38b04325-dockercfg-bwnc6\""
Sep 07 21:17:25.516466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.516406    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-38b04325\""
Sep 07 21:17:25.523092 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.523068    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"]
Sep 07 21:17:25.626300 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.626262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7w7q\" (UniqueName: \"kubernetes.io/projected/6b779927-e911-49ad-ae8d-c5faf132fcad-kube-api-access-w7w7q\") pod \"payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5\" (UID: \"6b779927-e911-49ad-ae8d-c5faf132fcad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:25.626485 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.626413    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b779927-e911-49ad-ae8d-c5faf132fcad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5\" (UID: \"6b779927-e911-49ad-ae8d-c5faf132fcad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:25.727730 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.727687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7w7q\" (UniqueName: \"kubernetes.io/projected/6b779927-e911-49ad-ae8d-c5faf132fcad-kube-api-access-w7w7q\") pod \"payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5\" (UID: \"6b779927-e911-49ad-ae8d-c5faf132fcad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:25.727875 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.727844    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b779927-e911-49ad-ae8d-c5faf132fcad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5\" (UID: \"6b779927-e911-49ad-ae8d-c5faf132fcad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:25.728560 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.728539    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b779927-e911-49ad-ae8d-c5faf132fcad-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5\" (UID: \"6b779927-e911-49ad-ae8d-c5faf132fcad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:25.736073 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.736049    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7w7q\" (UniqueName: \"kubernetes.io/projected/6b779927-e911-49ad-ae8d-c5faf132fcad-kube-api-access-w7w7q\") pod \"payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5\" (UID: \"6b779927-e911-49ad-ae8d-c5faf132fcad\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:25.771013 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.770940    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:17:25.825717 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.825681    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:25.828309 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828274    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istiod-ca-cert\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.828583 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828556    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:17:25.828685 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828604    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-data\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.828685 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828631    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-token\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.828685 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-envoy\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.828809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828692    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-socket\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.828809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828726    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-credential-socket\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.828809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828741    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zspd\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-kube-api-access-2zspd\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.828809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828776    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-certs\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.828809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.828798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-podinfo\") pod \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\" (UID: \"af565515-25ee-4fd9-91f3-2c499e1bb2e6\") "
Sep 07 21:17:25.829099 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.829071    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:25.829191 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.829169    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-socket\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:25.829191 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.829189    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istiod-ca-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:25.829360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.829211    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:25.829360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.829227    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:25.829517 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.829366    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:25.831431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.831396    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:17:25.831431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.831411    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-kube-api-access-2zspd" (OuterVolumeSpecName: "kube-api-access-2zspd") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "kube-api-access-2zspd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:17:25.831600 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.831481    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 21:17:25.831600 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.831593    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af565515-25ee-4fd9-91f3-2c499e1bb2e6" (UID: "af565515-25ee-4fd9-91f3-2c499e1bb2e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:25.930208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.930176    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-data\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:25.930208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.930204    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-token\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:25.930208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.930214    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-envoy\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:25.930418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.930223    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-credential-socket\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:25.930418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.930233    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zspd\" (UniqueName: \"kubernetes.io/projected/af565515-25ee-4fd9-91f3-2c499e1bb2e6-kube-api-access-2zspd\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:25.930418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.930241    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af565515-25ee-4fd9-91f3-2c499e1bb2e6-workload-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:25.930418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:25.930250    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af565515-25ee-4fd9-91f3-2c499e1bb2e6-istio-podinfo\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:26.097244 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.097132    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"]
Sep 07 21:17:26.099815 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:17:26.099784    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b779927_e911_49ad_ae8d_c5faf132fcad.slice/crio-39773638ac28db4364358113c80da20a8f6efd1f44b1913ab10087a2dcbaf84b WatchSource:0}: Error finding container 39773638ac28db4364358113c80da20a8f6efd1f44b1913ab10087a2dcbaf84b: Status 404 returned error can't find the container with id 39773638ac28db4364358113c80da20a8f6efd1f44b1913ab10087a2dcbaf84b
Sep 07 21:17:26.458841 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.458806    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5" event={"ID":"6b779927-e911-49ad-ae8d-c5faf132fcad","Type":"ContainerStarted","Data":"14306fdd1c1d1ab9d969b33326ef66df3d6157c59f5765eb685bb7d0a48f0d68"}
Sep 07 21:17:26.459058 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.458994    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:26.459058 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.459036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5" event={"ID":"6b779927-e911-49ad-ae8d-c5faf132fcad","Type":"ContainerStarted","Data":"39773638ac28db4364358113c80da20a8f6efd1f44b1913ab10087a2dcbaf84b"}
Sep 07 21:17:26.460705 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.460678    2565 generic.go:356] "Generic (PLEG): container finished" podID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerID="7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2" exitCode=0
Sep 07 21:17:26.460829 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.460711    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" event={"ID":"af565515-25ee-4fd9-91f3-2c499e1bb2e6","Type":"ContainerDied","Data":"7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2"}
Sep 07 21:17:26.460829 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.460723    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"
Sep 07 21:17:26.460829 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.460739    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf" event={"ID":"af565515-25ee-4fd9-91f3-2c499e1bb2e6","Type":"ContainerDied","Data":"7071b3f25157cdf3c5dd1585f6a4ea52a19a33c36837d5ba1114ad4822ca4205"}
Sep 07 21:17:26.460829 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.460760    2565 scope.go:117] "RemoveContainer" containerID="7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2"
Sep 07 21:17:26.469801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.469781    2565 scope.go:117] "RemoveContainer" containerID="ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6"
Sep 07 21:17:26.475990 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.475926    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5" podStartSLOduration=1.475910727 podStartE2EDuration="1.475910727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:17:26.474763005 +0000 UTC m=+938.556290347" watchObservedRunningTime="2026-09-07 21:17:26.475910727 +0000 UTC m=+938.557438069"
Sep 07 21:17:26.478301 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.478283    2565 scope.go:117] "RemoveContainer" containerID="7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2"
Sep 07 21:17:26.478556 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:17:26.478535    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2\": container with ID starting with 7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2 not found: ID does not exist" containerID="7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2"
Sep 07 21:17:26.478603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.478566    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2"} err="failed to get container status \"7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2\": rpc error: code = NotFound desc = could not find container \"7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2\": container with ID starting with 7ba7e8111fab9f30c1225bbaefe99f8ee08d7b64f640a1cd87c294eb3dd15bf2 not found: ID does not exist"
Sep 07 21:17:26.478603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.478587    2565 scope.go:117] "RemoveContainer" containerID="ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6"
Sep 07 21:17:26.478832 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:17:26.478814    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6\": container with ID starting with ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6 not found: ID does not exist" containerID="ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6"
Sep 07 21:17:26.478880 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.478838    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6"} err="failed to get container status \"ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6\": rpc error: code = NotFound desc = could not find container \"ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6\": container with ID starting with ba2dfeb8b8c5fafd891b809630ebb63e8fd67de6b486afc94251ec56a89dd1f6 not found: ID does not exist"
Sep 07 21:17:26.488700 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.488676    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"]
Sep 07 21:17:26.492340 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:26.492319    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b910426f-openshift-default-5cc74cf55b-4rpjf"]
Sep 07 21:17:28.364014 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:28.363981    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" path="/var/lib/kubelet/pods/af565515-25ee-4fd9-91f3-2c499e1bb2e6/volumes"
Sep 07 21:17:28.792807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:28.792708    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8" podUID="d1c5c8fd-e25f-4c2d-871d-baeaf7841849" containerName="main" containerID="cri-o://51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b" gracePeriod=45
Sep 07 21:17:29.047553 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.047497    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:17:29.160466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160434    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tmp-dir\") pod \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") "
Sep 07 21:17:29.160466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160469    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzp7t\" (UniqueName: \"kubernetes.io/projected/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-kube-api-access-hzp7t\") pod \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") "
Sep 07 21:17:29.160716 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-model-cache\") pod \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") "
Sep 07 21:17:29.160716 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160532    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-dshm\") pod \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") "
Sep 07 21:17:29.160716 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160569    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tls-certs\") pod \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") "
Sep 07 21:17:29.160716 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160654    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-home\") pod \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\" (UID: \"d1c5c8fd-e25f-4c2d-871d-baeaf7841849\") "
Sep 07 21:17:29.160915 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160709    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d1c5c8fd-e25f-4c2d-871d-baeaf7841849" (UID: "d1c5c8fd-e25f-4c2d-871d-baeaf7841849"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:29.160915 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160838    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-model-cache" (OuterVolumeSpecName: "model-cache") pod "d1c5c8fd-e25f-4c2d-871d-baeaf7841849" (UID: "d1c5c8fd-e25f-4c2d-871d-baeaf7841849"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:29.161023 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.160953    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-home" (OuterVolumeSpecName: "home") pod "d1c5c8fd-e25f-4c2d-871d-baeaf7841849" (UID: "d1c5c8fd-e25f-4c2d-871d-baeaf7841849"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:29.161174 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.161155    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:29.161231 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.161178    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:29.161231 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.161191    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:29.162917 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.162894    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-kube-api-access-hzp7t" (OuterVolumeSpecName: "kube-api-access-hzp7t") pod "d1c5c8fd-e25f-4c2d-871d-baeaf7841849" (UID: "d1c5c8fd-e25f-4c2d-871d-baeaf7841849"). InnerVolumeSpecName "kube-api-access-hzp7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:17:29.163033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.162921    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-dshm" (OuterVolumeSpecName: "dshm") pod "d1c5c8fd-e25f-4c2d-871d-baeaf7841849" (UID: "d1c5c8fd-e25f-4c2d-871d-baeaf7841849"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:29.163033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.162922    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d1c5c8fd-e25f-4c2d-871d-baeaf7841849" (UID: "d1c5c8fd-e25f-4c2d-871d-baeaf7841849"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:17:29.261848 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.261809    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:29.261848 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.261846    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:29.261848 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.261858    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzp7t\" (UniqueName: \"kubernetes.io/projected/d1c5c8fd-e25f-4c2d-871d-baeaf7841849-kube-api-access-hzp7t\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:29.480232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.480199    2565 generic.go:356] "Generic (PLEG): container finished" podID="d1c5c8fd-e25f-4c2d-871d-baeaf7841849" containerID="51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b" exitCode=0
Sep 07 21:17:29.480628 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.480250    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"
Sep 07 21:17:29.480628 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.480276    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8" event={"ID":"d1c5c8fd-e25f-4c2d-871d-baeaf7841849","Type":"ContainerDied","Data":"51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b"}
Sep 07 21:17:29.480628 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.480316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8" event={"ID":"d1c5c8fd-e25f-4c2d-871d-baeaf7841849","Type":"ContainerDied","Data":"0574f1d4f600715cf1bfcf83c5508af00afc0b3239e406f49b0900e70dbb1529"}
Sep 07 21:17:29.480628 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.480334    2565 scope.go:117] "RemoveContainer" containerID="51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b"
Sep 07 21:17:29.491710 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.491654    2565 scope.go:117] "RemoveContainer" containerID="51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b"
Sep 07 21:17:29.492241 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:17:29.492214    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b\": container with ID starting with 51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b not found: ID does not exist" containerID="51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b"
Sep 07 21:17:29.492359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.492250    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b"} err="failed to get container status \"51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b\": rpc error: code = NotFound desc = could not find container \"51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b\": container with ID starting with 51db9e91abc6e8a39fdb84a92ff47e16ab668053805aa4a5804173a705b1694b not found: ID does not exist"
Sep 07 21:17:29.501665 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.501644    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"]
Sep 07 21:17:29.510755 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:29.510727    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-9e7d7ca5-kserve-5dfb4b498d-wc7d8"]
Sep 07 21:17:30.363210 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:30.363177    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c5c8fd-e25f-4c2d-871d-baeaf7841849" path="/var/lib/kubelet/pods/d1c5c8fd-e25f-4c2d-871d-baeaf7841849/volumes"
Sep 07 21:17:31.462105 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.462068    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s" podUID="f2601f5b-b506-4d2b-9ca5-7d17ddacd654" containerName="main" containerID="cri-o://d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be" gracePeriod=45
Sep 07 21:17:31.711860 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.711835    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:17:31.784764 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.784694    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tls-certs\") pod \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") "
Sep 07 21:17:31.784764 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.784744    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c7sj\" (UniqueName: \"kubernetes.io/projected/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-kube-api-access-8c7sj\") pod \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") "
Sep 07 21:17:31.785001 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.784772    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-dshm\") pod \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") "
Sep 07 21:17:31.785001 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.784880    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-model-cache\") pod \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") "
Sep 07 21:17:31.785001 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.784907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tmp-dir\") pod \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") "
Sep 07 21:17:31.785001 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.784955    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-home\") pod \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\" (UID: \"f2601f5b-b506-4d2b-9ca5-7d17ddacd654\") "
Sep 07 21:17:31.785230 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.785183    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2601f5b-b506-4d2b-9ca5-7d17ddacd654" (UID: "f2601f5b-b506-4d2b-9ca5-7d17ddacd654"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:31.785230 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.785210    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2601f5b-b506-4d2b-9ca5-7d17ddacd654" (UID: "f2601f5b-b506-4d2b-9ca5-7d17ddacd654"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:31.785338 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.785289    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-home" (OuterVolumeSpecName: "home") pod "f2601f5b-b506-4d2b-9ca5-7d17ddacd654" (UID: "f2601f5b-b506-4d2b-9ca5-7d17ddacd654"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:31.785418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.785402    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:31.785478 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.785419    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:31.785478 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.785428    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:31.786936 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.786915    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-kube-api-access-8c7sj" (OuterVolumeSpecName: "kube-api-access-8c7sj") pod "f2601f5b-b506-4d2b-9ca5-7d17ddacd654" (UID: "f2601f5b-b506-4d2b-9ca5-7d17ddacd654"). InnerVolumeSpecName "kube-api-access-8c7sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:17:31.787271 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.787258    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-dshm" (OuterVolumeSpecName: "dshm") pod "f2601f5b-b506-4d2b-9ca5-7d17ddacd654" (UID: "f2601f5b-b506-4d2b-9ca5-7d17ddacd654"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:17:31.787322 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.787300    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2601f5b-b506-4d2b-9ca5-7d17ddacd654" (UID: "f2601f5b-b506-4d2b-9ca5-7d17ddacd654"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:17:31.886849 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.886819    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:31.886849 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.886845    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c7sj\" (UniqueName: \"kubernetes.io/projected/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-kube-api-access-8c7sj\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:31.886849 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:31.886853    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2601f5b-b506-4d2b-9ca5-7d17ddacd654-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:32.496061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.496024    2565 generic.go:356] "Generic (PLEG): container finished" podID="f2601f5b-b506-4d2b-9ca5-7d17ddacd654" containerID="d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be" exitCode=0
Sep 07 21:17:32.496061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.496062    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"
Sep 07 21:17:32.496500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.496109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s" event={"ID":"f2601f5b-b506-4d2b-9ca5-7d17ddacd654","Type":"ContainerDied","Data":"d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be"}
Sep 07 21:17:32.496500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.496148    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s" event={"ID":"f2601f5b-b506-4d2b-9ca5-7d17ddacd654","Type":"ContainerDied","Data":"ec1d39eae00996bf429532346d172bd855c5b65fce99d6ec50d21642ada0d2ca"}
Sep 07 21:17:32.496500 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.496175    2565 scope.go:117] "RemoveContainer" containerID="d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be"
Sep 07 21:17:32.504587 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.504570    2565 scope.go:117] "RemoveContainer" containerID="d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be"
Sep 07 21:17:32.504856 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:17:32.504839    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be\": container with ID starting with d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be not found: ID does not exist" containerID="d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be"
Sep 07 21:17:32.504900 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.504864    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be"} err="failed to get container status \"d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be\": rpc error: code = NotFound desc = could not find container \"d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be\": container with ID starting with d1bb663d223ea2156d4e339dfb0660ab9f4d4ba078c9b6d6b37bb80f8c1e83be not found: ID does not exist"
Sep 07 21:17:32.511021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.511000    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"]
Sep 07 21:17:32.513400 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.513380    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-9e7d7ca5-kserve-8db57599d-lnp5s"]
Sep 07 21:17:32.861278 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:32.861246    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:17:34.364212 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.364180    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2601f5b-b506-4d2b-9ca5-7d17ddacd654" path="/var/lib/kubelet/pods/f2601f5b-b506-4d2b-9ca5-7d17ddacd654/volumes"
Sep 07 21:17:34.804512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804434    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"]
Sep 07 21:17:34.804791 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804780    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2601f5b-b506-4d2b-9ca5-7d17ddacd654" containerName="main"
Sep 07 21:17:34.804835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804793    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2601f5b-b506-4d2b-9ca5-7d17ddacd654" containerName="main"
Sep 07 21:17:34.804835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804805    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c5c8fd-e25f-4c2d-871d-baeaf7841849" containerName="main"
Sep 07 21:17:34.804835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804810    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c5c8fd-e25f-4c2d-871d-baeaf7841849" containerName="main"
Sep 07 21:17:34.804835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804822    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy"
Sep 07 21:17:34.804835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804828    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy"
Sep 07 21:17:34.805007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804841    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy"
Sep 07 21:17:34.805007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804846    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy"
Sep 07 21:17:34.805007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804908    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c5c8fd-e25f-4c2d-871d-baeaf7841849" containerName="main"
Sep 07 21:17:34.805007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804916    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy"
Sep 07 21:17:34.805007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804924    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="af565515-25ee-4fd9-91f3-2c499e1bb2e6" containerName="istio-proxy"
Sep 07 21:17:34.805007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.804930    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2601f5b-b506-4d2b-9ca5-7d17ddacd654" containerName="main"
Sep 07 21:17:34.809442 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.809416    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.812248 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.812220    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f447ae-openshift-default-dockercfg-cvnpk\""
Sep 07 21:17:34.820422 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.820401    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"]
Sep 07 21:17:34.916161 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916120    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ce3ecc-7014-4e36-884f-6dca0251343f-istiod-ca-cert\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.916328 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916176    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-podinfo\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.916328 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916203    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-token\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.916328 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-certs\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.916472 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-socket\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.916472 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-envoy\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.916472 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916395    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-credential-socket\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.916472 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcds\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-kube-api-access-kgcds\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:34.916608 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:34.916491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-data\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017122 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017086    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-envoy\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017122 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017126    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-credential-socket\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcds\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-kube-api-access-kgcds\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-data\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017197    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ce3ecc-7014-4e36-884f-6dca0251343f-istiod-ca-cert\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-podinfo\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-token\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017264    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-certs\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017331    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-socket\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017747 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017619    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-credential-socket\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017747 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017688    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-socket\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017850 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017828    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-data\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.017897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.017867    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-certs\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.018023 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.018006    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ce3ecc-7014-4e36-884f-6dca0251343f-istiod-ca-cert\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.019434 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.019414    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-envoy\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.019685 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.019664    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-podinfo\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.024922 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.024895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-token\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.025085 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.025067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcds\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-kube-api-access-kgcds\") pod \"e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") " pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.120166 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.120142    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:35.392095 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.392074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"]
Sep 07 21:17:35.394240 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:17:35.394213    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ce3ecc_7014_4e36_884f_6dca0251343f.slice/crio-757a70643213c164907cd7d2cace35baf318ab8ce1ebf51acfd50b939a545098 WatchSource:0}: Error finding container 757a70643213c164907cd7d2cace35baf318ab8ce1ebf51acfd50b939a545098: Status 404 returned error can't find the container with id 757a70643213c164907cd7d2cace35baf318ab8ce1ebf51acfd50b939a545098
Sep 07 21:17:35.396463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.396432    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:17:35.396537 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.396495    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:17:35.396537 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.396525    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:17:35.522175 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.522141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" event={"ID":"68ce3ecc-7014-4e36-884f-6dca0251343f","Type":"ContainerStarted","Data":"66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79"}
Sep 07 21:17:35.522338 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.522204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" event={"ID":"68ce3ecc-7014-4e36-884f-6dca0251343f","Type":"ContainerStarted","Data":"757a70643213c164907cd7d2cace35baf318ab8ce1ebf51acfd50b939a545098"}
Sep 07 21:17:35.543332 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:35.543286    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" podStartSLOduration=1.543269765 podStartE2EDuration="1.543269765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:17:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:17:35.542213786 +0000 UTC m=+947.623741140" watchObservedRunningTime="2026-09-07 21:17:35.543269765 +0000 UTC m=+947.624797105"
Sep 07 21:17:36.120631 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:36.120594    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:36.125811 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:36.125786    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:36.527281 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:36.527203    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:36.528222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:36.528199    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:17:37.470693 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:37.470657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:39.330926 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.330888    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"]
Sep 07 21:17:39.334530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.334510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:39.336885 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.336861    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f447ae-dockercfg-rx9xq\""
Sep 07 21:17:39.337011 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.336950    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f447ae\""
Sep 07 21:17:39.345685 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.345660    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"]
Sep 07 21:17:39.376984 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.376938    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"]
Sep 07 21:17:39.380198 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.380179    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:39.392644 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.392625    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"]
Sep 07 21:17:39.458207 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.458173    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p794\" (UniqueName: \"kubernetes.io/projected/dff48924-2038-4fbc-80cc-8e29f755b98b-kube-api-access-4p794\") pod \"payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h\" (UID: \"dff48924-2038-4fbc-80cc-8e29f755b98b\") " pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:39.458374 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.458268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff48924-2038-4fbc-80cc-8e29f755b98b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h\" (UID: \"dff48924-2038-4fbc-80cc-8e29f755b98b\") " pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:39.558936 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.558900    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff48924-2038-4fbc-80cc-8e29f755b98b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h\" (UID: \"dff48924-2038-4fbc-80cc-8e29f755b98b\") " pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:39.559154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.559015    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p794\" (UniqueName: \"kubernetes.io/projected/dff48924-2038-4fbc-80cc-8e29f755b98b-kube-api-access-4p794\") pod \"payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h\" (UID: \"dff48924-2038-4fbc-80cc-8e29f755b98b\") " pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:39.559154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.559054    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtlh\" (UniqueName: \"kubernetes.io/projected/6e28e179-25f6-4bf5-b769-e3c30da4480a-kube-api-access-qqtlh\") pod \"payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm\" (UID: \"6e28e179-25f6-4bf5-b769-e3c30da4480a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:39.559154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.559102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e28e179-25f6-4bf5-b769-e3c30da4480a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm\" (UID: \"6e28e179-25f6-4bf5-b769-e3c30da4480a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:39.559614 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.559593    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff48924-2038-4fbc-80cc-8e29f755b98b-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h\" (UID: \"dff48924-2038-4fbc-80cc-8e29f755b98b\") " pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:39.568625 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.568602    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p794\" (UniqueName: \"kubernetes.io/projected/dff48924-2038-4fbc-80cc-8e29f755b98b-kube-api-access-4p794\") pod \"payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h\" (UID: \"dff48924-2038-4fbc-80cc-8e29f755b98b\") " pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:39.646089 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.646062    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:39.660122 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.660098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtlh\" (UniqueName: \"kubernetes.io/projected/6e28e179-25f6-4bf5-b769-e3c30da4480a-kube-api-access-qqtlh\") pod \"payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm\" (UID: \"6e28e179-25f6-4bf5-b769-e3c30da4480a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:39.660232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.660137    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e28e179-25f6-4bf5-b769-e3c30da4480a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm\" (UID: \"6e28e179-25f6-4bf5-b769-e3c30da4480a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:39.661012 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.660933    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e28e179-25f6-4bf5-b769-e3c30da4480a-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm\" (UID: \"6e28e179-25f6-4bf5-b769-e3c30da4480a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:39.670153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.670130    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtlh\" (UniqueName: \"kubernetes.io/projected/6e28e179-25f6-4bf5-b769-e3c30da4480a-kube-api-access-qqtlh\") pod \"payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm\" (UID: \"6e28e179-25f6-4bf5-b769-e3c30da4480a\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:39.689095 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.689071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:39.916146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.916118    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"]
Sep 07 21:17:39.917524 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:17:39.917497    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff48924_2038_4fbc_80cc_8e29f755b98b.slice/crio-974310f274c84a24f1832ade6f6be2356edec29e6ac7ed8ac3851c120a3d0d84 WatchSource:0}: Error finding container 974310f274c84a24f1832ade6f6be2356edec29e6ac7ed8ac3851c120a3d0d84: Status 404 returned error can't find the container with id 974310f274c84a24f1832ade6f6be2356edec29e6ac7ed8ac3851c120a3d0d84
Sep 07 21:17:39.966867 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:39.966845    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"]
Sep 07 21:17:39.969949 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:17:39.969923    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e28e179_25f6_4bf5_b769_e3c30da4480a.slice/crio-ccddad589eb271b516c3b0a769913d17c44d5386ec39882c6a15ad1f5c05c995 WatchSource:0}: Error finding container ccddad589eb271b516c3b0a769913d17c44d5386ec39882c6a15ad1f5c05c995: Status 404 returned error can't find the container with id ccddad589eb271b516c3b0a769913d17c44d5386ec39882c6a15ad1f5c05c995
Sep 07 21:17:40.548101 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:40.548058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h" event={"ID":"dff48924-2038-4fbc-80cc-8e29f755b98b","Type":"ContainerStarted","Data":"8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9"}
Sep 07 21:17:40.548535 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:40.548125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h" event={"ID":"dff48924-2038-4fbc-80cc-8e29f755b98b","Type":"ContainerStarted","Data":"974310f274c84a24f1832ade6f6be2356edec29e6ac7ed8ac3851c120a3d0d84"}
Sep 07 21:17:40.548535 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:40.548186    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:40.549569 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:40.549548    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm" event={"ID":"6e28e179-25f6-4bf5-b769-e3c30da4480a","Type":"ContainerStarted","Data":"08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02"}
Sep 07 21:17:40.549683 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:40.549583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm" event={"ID":"6e28e179-25f6-4bf5-b769-e3c30da4480a","Type":"ContainerStarted","Data":"ccddad589eb271b516c3b0a769913d17c44d5386ec39882c6a15ad1f5c05c995"}
Sep 07 21:17:40.549683 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:40.549667    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:40.565262 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:40.565213    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h" podStartSLOduration=1.565199512 podStartE2EDuration="1.565199512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:17:40.563895647 +0000 UTC m=+952.645422989" watchObservedRunningTime="2026-09-07 21:17:40.565199512 +0000 UTC m=+952.646726843"
Sep 07 21:17:40.579913 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:40.579867    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm" podStartSLOduration=1.579854121 podStartE2EDuration="1.579854121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:17:40.578581607 +0000 UTC m=+952.660108948" watchObservedRunningTime="2026-09-07 21:17:40.579854121 +0000 UTC m=+952.661381461"
Sep 07 21:17:48.226365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.226330    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"]
Sep 07 21:17:48.229597 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.229582    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"
Sep 07 21:17:48.231882 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.231858    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:17:48.232035 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.231862    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:17:48.235289 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.235207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea89184d-00f1-44c5-9f2c-32afdc70453c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-38b04325-8x4bk\" (UID: \"ea89184d-00f1-44c5-9f2c-32afdc70453c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"
Sep 07 21:17:48.236471 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.236450    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"]
Sep 07 21:17:48.335534 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.335504    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea89184d-00f1-44c5-9f2c-32afdc70453c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-38b04325-8x4bk\" (UID: \"ea89184d-00f1-44c5-9f2c-32afdc70453c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"
Sep 07 21:17:48.337530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.337507    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:17:48.347021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.347001    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea89184d-00f1-44c5-9f2c-32afdc70453c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-38b04325-8x4bk\" (UID: \"ea89184d-00f1-44c5-9f2c-32afdc70453c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"
Sep 07 21:17:48.543291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.543228    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:17:48.551954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.551935    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"
Sep 07 21:17:48.829407 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:48.829384    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"]
Sep 07 21:17:48.831648 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:17:48.831613    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea89184d_00f1_44c5_9f2c_32afdc70453c.slice/crio-e676c27ede39b0b079070a494bf7d42367c9798216ace496cf839ae83c475770 WatchSource:0}: Error finding container e676c27ede39b0b079070a494bf7d42367c9798216ace496cf839ae83c475770: Status 404 returned error can't find the container with id e676c27ede39b0b079070a494bf7d42367c9798216ace496cf839ae83c475770
Sep 07 21:17:49.153650 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:49.153615    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:17:49.598007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:49.597947    2565 generic.go:356] "Generic (PLEG): container finished" podID="ea89184d-00f1-44c5-9f2c-32afdc70453c" containerID="6ca45a5d6a1a7bbd875321f805b09feae9ce09e518c14e01c098bcd120b9b8fd" exitCode=0
Sep 07 21:17:49.598366 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:49.598036    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk" event={"ID":"ea89184d-00f1-44c5-9f2c-32afdc70453c","Type":"ContainerDied","Data":"6ca45a5d6a1a7bbd875321f805b09feae9ce09e518c14e01c098bcd120b9b8fd"}
Sep 07 21:17:49.598366 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:49.598087    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk" event={"ID":"ea89184d-00f1-44c5-9f2c-32afdc70453c","Type":"ContainerStarted","Data":"e676c27ede39b0b079070a494bf7d42367c9798216ace496cf839ae83c475770"}
Sep 07 21:17:50.722120 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:50.722088    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"
Sep 07 21:17:50.858838 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:50.858295    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea89184d-00f1-44c5-9f2c-32afdc70453c-maas-api-service-ca\") pod \"ea89184d-00f1-44c5-9f2c-32afdc70453c\" (UID: \"ea89184d-00f1-44c5-9f2c-32afdc70453c\") "
Sep 07 21:17:50.859233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:50.859191    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea89184d-00f1-44c5-9f2c-32afdc70453c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea89184d-00f1-44c5-9f2c-32afdc70453c" (UID: "ea89184d-00f1-44c5-9f2c-32afdc70453c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:17:50.959807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:50.959765    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea89184d-00f1-44c5-9f2c-32afdc70453c-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:51.557108 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:51.557077    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:17:51.557283 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:51.557146    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:17:51.609912 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:51.609880    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk" event={"ID":"ea89184d-00f1-44c5-9f2c-32afdc70453c","Type":"ContainerDied","Data":"e676c27ede39b0b079070a494bf7d42367c9798216ace496cf839ae83c475770"}
Sep 07 21:17:51.610083 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:51.609913    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"
Sep 07 21:17:51.610083 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:51.609950    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e676c27ede39b0b079070a494bf7d42367c9798216ace496cf839ae83c475770"
Sep 07 21:17:55.351237 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.351196    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"]
Sep 07 21:17:55.352078 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.352046    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5" podUID="6b779927-e911-49ad-ae8d-c5faf132fcad" containerName="payload-pre-processing" containerID="cri-o://14306fdd1c1d1ab9d969b33326ef66df3d6157c59f5765eb685bb7d0a48f0d68" gracePeriod=30
Sep 07 21:17:55.504227 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.504194    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"]
Sep 07 21:17:55.511342 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.511314    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-38b04325-8x4bk"]
Sep 07 21:17:55.634433 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.634309    2565 generic.go:356] "Generic (PLEG): container finished" podID="6b779927-e911-49ad-ae8d-c5faf132fcad" containerID="14306fdd1c1d1ab9d969b33326ef66df3d6157c59f5765eb685bb7d0a48f0d68" exitCode=0
Sep 07 21:17:55.634433 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.634364    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5" event={"ID":"6b779927-e911-49ad-ae8d-c5faf132fcad","Type":"ContainerDied","Data":"14306fdd1c1d1ab9d969b33326ef66df3d6157c59f5765eb685bb7d0a48f0d68"}
Sep 07 21:17:55.692945 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.692839    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:55.719676 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.719651    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w7w7q\" (UniqueName: \"kubernetes.io/projected/6b779927-e911-49ad-ae8d-c5faf132fcad-kube-api-access-w7w7q\") pod \"6b779927-e911-49ad-ae8d-c5faf132fcad\" (UID: \"6b779927-e911-49ad-ae8d-c5faf132fcad\") "
Sep 07 21:17:55.719837 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.719745    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b779927-e911-49ad-ae8d-c5faf132fcad-plugins-config-volume\") pod \"6b779927-e911-49ad-ae8d-c5faf132fcad\" (UID: \"6b779927-e911-49ad-ae8d-c5faf132fcad\") "
Sep 07 21:17:55.720203 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.720176    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b779927-e911-49ad-ae8d-c5faf132fcad-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6b779927-e911-49ad-ae8d-c5faf132fcad" (UID: "6b779927-e911-49ad-ae8d-c5faf132fcad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:17:55.721623 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.721601    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b779927-e911-49ad-ae8d-c5faf132fcad-kube-api-access-w7w7q" (OuterVolumeSpecName: "kube-api-access-w7w7q") pod "6b779927-e911-49ad-ae8d-c5faf132fcad" (UID: "6b779927-e911-49ad-ae8d-c5faf132fcad"). InnerVolumeSpecName "kube-api-access-w7w7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:17:55.820426 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.820395    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7w7q\" (UniqueName: \"kubernetes.io/projected/6b779927-e911-49ad-ae8d-c5faf132fcad-kube-api-access-w7w7q\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:55.820426 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:55.820422    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b779927-e911-49ad-ae8d-c5faf132fcad-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:17:56.365614 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:56.365583    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea89184d-00f1-44c5-9f2c-32afdc70453c" path="/var/lib/kubelet/pods/ea89184d-00f1-44c5-9f2c-32afdc70453c/volumes"
Sep 07 21:17:56.640420 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:56.640384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5" event={"ID":"6b779927-e911-49ad-ae8d-c5faf132fcad","Type":"ContainerDied","Data":"39773638ac28db4364358113c80da20a8f6efd1f44b1913ab10087a2dcbaf84b"}
Sep 07 21:17:56.640420 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:56.640412    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"
Sep 07 21:17:56.640661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:56.640481    2565 scope.go:117] "RemoveContainer" containerID="14306fdd1c1d1ab9d969b33326ef66df3d6157c59f5765eb685bb7d0a48f0d68"
Sep 07 21:17:56.655185 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:56.655160    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"]
Sep 07 21:17:56.657485 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:56.657463    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-38b04325-5fc79677c4-qmbh5"]
Sep 07 21:17:58.364079 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:58.364045    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b779927-e911-49ad-ae8d-c5faf132fcad" path="/var/lib/kubelet/pods/6b779927-e911-49ad-ae8d-c5faf132fcad/volumes"
Sep 07 21:17:59.657870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:59.657839    2565 generic.go:356] "Generic (PLEG): container finished" podID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerID="66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79" exitCode=0
Sep 07 21:17:59.658509 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:59.657885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" event={"ID":"68ce3ecc-7014-4e36-884f-6dca0251343f","Type":"ContainerDied","Data":"66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79"}
Sep 07 21:17:59.658509 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:59.658247    2565 scope.go:117] "RemoveContainer" containerID="66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79"
Sep 07 21:17:59.659404 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:59.659374    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:17:59.659503 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:59.659459    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:17:59.659553 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:17:59.659512    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 07 21:18:00.668948 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:00.668896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" event={"ID":"68ce3ecc-7014-4e36-884f-6dca0251343f","Type":"ContainerStarted","Data":"777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3"}
Sep 07 21:18:01.120702 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:01.120669    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:18:01.120930 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:01.120909    2565 patch_prober.go:28] interesting pod/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 21:18:01.121005 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:01.120984    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 21:18:02.120615 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.120581    2565 patch_prober.go:28] interesting pod/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 21:18:02.121044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.120662    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 21:18:02.848291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.848258    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"]
Sep 07 21:18:02.848616 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.848603    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea89184d-00f1-44c5-9f2c-32afdc70453c" containerName="revoke-keys"
Sep 07 21:18:02.848681 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.848620    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea89184d-00f1-44c5-9f2c-32afdc70453c" containerName="revoke-keys"
Sep 07 21:18:02.848681 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.848636    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b779927-e911-49ad-ae8d-c5faf132fcad" containerName="payload-pre-processing"
Sep 07 21:18:02.848681 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.848644    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b779927-e911-49ad-ae8d-c5faf132fcad" containerName="payload-pre-processing"
Sep 07 21:18:02.848845 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.848746    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea89184d-00f1-44c5-9f2c-32afdc70453c" containerName="revoke-keys"
Sep 07 21:18:02.848845 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.848760    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b779927-e911-49ad-ae8d-c5faf132fcad" containerName="payload-pre-processing"
Sep 07 21:18:02.851909 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.851887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.855154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.855033    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3102b2\"/\"kube-root-ca.crt\""
Sep 07 21:18:02.855154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.855086    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3102b2\"/\"test-model-3102b2-kserve-self-signed-certs\""
Sep 07 21:18:02.855337 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.855209    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3102b2\"/\"openshift-service-ca.crt\""
Sep 07 21:18:02.855337 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.855301    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3102b2\"/\"default-dockercfg-qmg52\""
Sep 07 21:18:02.876738 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.876709    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"]
Sep 07 21:18:02.894794 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.894760    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-model-cache\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.894943 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.894887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tls-certs\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.895033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.895014    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m972c\" (UniqueName: \"kubernetes.io/projected/c9998df7-d3ec-4cad-8d61-e9be1abde86b-kube-api-access-m972c\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.895079 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.895039    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tmp-dir\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.895164 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.895140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-home\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.895228 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.895183    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-dshm\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.995626 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.995583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-dshm\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.995821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.995636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-model-cache\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.995821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.995668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tls-certs\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.995821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.995712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m972c\" (UniqueName: \"kubernetes.io/projected/c9998df7-d3ec-4cad-8d61-e9be1abde86b-kube-api-access-m972c\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.995821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.995732    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tmp-dir\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.995821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.995759    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-home\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.996196 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.996172    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-home\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.996270 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.996208    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tmp-dir\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.996363 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.996344    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-model-cache\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.998042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.998023    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-dshm\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:02.998200 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:02.998184    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tls-certs\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:03.003426 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:03.003404    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m972c\" (UniqueName: \"kubernetes.io/projected/c9998df7-d3ec-4cad-8d61-e9be1abde86b-kube-api-access-m972c\") pod \"test-model-3102b2-kserve-67556d4f49-rpksv\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") " pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:03.120923 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:03.120834    2565 patch_prober.go:28] interesting pod/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 21:18:03.120923 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:03.120889    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 21:18:03.181084 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:03.181051    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:03.452736 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:03.452710    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"]
Sep 07 21:18:03.458179 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:18:03.458121    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9998df7_d3ec_4cad_8d61_e9be1abde86b.slice/crio-6abcbd1f84b43251cd04996936567218b8506049169dffe4d360a3dfc5ec3c85 WatchSource:0}: Error finding container 6abcbd1f84b43251cd04996936567218b8506049169dffe4d360a3dfc5ec3c85: Status 404 returned error can't find the container with id 6abcbd1f84b43251cd04996936567218b8506049169dffe4d360a3dfc5ec3c85
Sep 07 21:18:03.690487 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:03.690455    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv" event={"ID":"c9998df7-d3ec-4cad-8d61-e9be1abde86b","Type":"ContainerStarted","Data":"38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736"}
Sep 07 21:18:03.690707 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:03.690692    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv" event={"ID":"c9998df7-d3ec-4cad-8d61-e9be1abde86b","Type":"ContainerStarted","Data":"6abcbd1f84b43251cd04996936567218b8506049169dffe4d360a3dfc5ec3c85"}
Sep 07 21:18:03.707076 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:03.706978    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv" podStartSLOduration=1.706951164 podStartE2EDuration="1.706951164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:18:03.704796179 +0000 UTC m=+975.786323522" watchObservedRunningTime="2026-09-07 21:18:03.706951164 +0000 UTC m=+975.788478504"
Sep 07 21:18:04.120784 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:04.120744    2565 patch_prober.go:28] interesting pod/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 21:18:04.120954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:04.120813    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 21:18:05.121187 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:05.121148    2565 patch_prober.go:28] interesting pod/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body=
Sep 07 21:18:05.121635 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:05.121207    2565 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused"
Sep 07 21:18:06.125164 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:06.125126    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:18:06.125559 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:06.125538    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:18:06.126318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:06.126298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:18:07.959264 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:07.959230    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:18:13.181517 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:13.181429    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:13.181517 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:13.181483    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:13.194253 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:13.194228    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:13.752105 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:13.752077    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:18:23.958420 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:23.958384    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:18:41.089701 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.089664    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"]
Sep 07 21:18:41.094036 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.094019    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"
Sep 07 21:18:41.096612 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.096589    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:18:41.096612 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.096607    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:18:41.100161 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.100140    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"]
Sep 07 21:18:41.145120 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.145093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3846750b-c3e5-47ab-86bd-7a32c1111756-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f447ae-l7jng\" (UID: \"3846750b-c3e5-47ab-86bd-7a32c1111756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"
Sep 07 21:18:41.246179 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.246151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3846750b-c3e5-47ab-86bd-7a32c1111756-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f447ae-l7jng\" (UID: \"3846750b-c3e5-47ab-86bd-7a32c1111756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"
Sep 07 21:18:41.246714 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.246696    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3846750b-c3e5-47ab-86bd-7a32c1111756-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f447ae-l7jng\" (UID: \"3846750b-c3e5-47ab-86bd-7a32c1111756\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"
Sep 07 21:18:41.407437 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.407365    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"
Sep 07 21:18:41.675774 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.675622    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"]
Sep 07 21:18:41.678202 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:18:41.678173    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3846750b_c3e5_47ab_86bd_7a32c1111756.slice/crio-5466aab7a94cb92d17139af05ff2bcf2d406a2de595978f85040183597b29be7 WatchSource:0}: Error finding container 5466aab7a94cb92d17139af05ff2bcf2d406a2de595978f85040183597b29be7: Status 404 returned error can't find the container with id 5466aab7a94cb92d17139af05ff2bcf2d406a2de595978f85040183597b29be7
Sep 07 21:18:41.882401 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:41.882370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng" event={"ID":"3846750b-c3e5-47ab-86bd-7a32c1111756","Type":"ContainerStarted","Data":"5466aab7a94cb92d17139af05ff2bcf2d406a2de595978f85040183597b29be7"}
Sep 07 21:18:42.888690 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:42.888659    2565 generic.go:356] "Generic (PLEG): container finished" podID="3846750b-c3e5-47ab-86bd-7a32c1111756" containerID="17505dcd7cbe3ec8e345b4daf60cc52c669ba17bf25d05638062343631ae9f7a" exitCode=0
Sep 07 21:18:42.889088 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:42.888741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng" event={"ID":"3846750b-c3e5-47ab-86bd-7a32c1111756","Type":"ContainerDied","Data":"17505dcd7cbe3ec8e345b4daf60cc52c669ba17bf25d05638062343631ae9f7a"}
Sep 07 21:18:44.023329 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:44.023302    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"
Sep 07 21:18:44.069425 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:44.069392    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3846750b-c3e5-47ab-86bd-7a32c1111756-maas-api-service-ca\") pod \"3846750b-c3e5-47ab-86bd-7a32c1111756\" (UID: \"3846750b-c3e5-47ab-86bd-7a32c1111756\") "
Sep 07 21:18:44.069732 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:44.069710    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3846750b-c3e5-47ab-86bd-7a32c1111756-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3846750b-c3e5-47ab-86bd-7a32c1111756" (UID: "3846750b-c3e5-47ab-86bd-7a32c1111756"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:18:44.170073 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:44.170003    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3846750b-c3e5-47ab-86bd-7a32c1111756-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:18:44.851668 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:44.851633    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:18:44.899316 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:44.899284    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng" event={"ID":"3846750b-c3e5-47ab-86bd-7a32c1111756","Type":"ContainerDied","Data":"5466aab7a94cb92d17139af05ff2bcf2d406a2de595978f85040183597b29be7"}
Sep 07 21:18:44.899316 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:44.899311    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"
Sep 07 21:18:44.899511 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:44.899328    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5466aab7a94cb92d17139af05ff2bcf2d406a2de595978f85040183597b29be7"
Sep 07 21:18:51.567545 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.567506    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"]
Sep 07 21:18:51.568073 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.567799    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h" podUID="dff48924-2038-4fbc-80cc-8e29f755b98b" containerName="payload-processing" containerID="cri-o://8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9" gracePeriod=30
Sep 07 21:18:51.571411 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.571385    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"]
Sep 07 21:18:51.571665 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.571611    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm" podUID="6e28e179-25f6-4bf5-b769-e3c30da4480a" containerName="payload-pre-processing" containerID="cri-o://08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02" gracePeriod=30
Sep 07 21:18:51.720380 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.720153    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"]
Sep 07 21:18:51.722341 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.722319    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f447ae-l7jng"]
Sep 07 21:18:51.829752 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.829695    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:18:51.852777 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.852752    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:18:51.936069 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936024    2565 generic.go:356] "Generic (PLEG): container finished" podID="dff48924-2038-4fbc-80cc-8e29f755b98b" containerID="8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9" exitCode=0
Sep 07 21:18:51.936069 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936072    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"
Sep 07 21:18:51.936318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936097    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qqtlh\" (UniqueName: \"kubernetes.io/projected/6e28e179-25f6-4bf5-b769-e3c30da4480a-kube-api-access-qqtlh\") pod \"6e28e179-25f6-4bf5-b769-e3c30da4480a\" (UID: \"6e28e179-25f6-4bf5-b769-e3c30da4480a\") "
Sep 07 21:18:51.936318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h" event={"ID":"dff48924-2038-4fbc-80cc-8e29f755b98b","Type":"ContainerDied","Data":"8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9"}
Sep 07 21:18:51.936318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936151    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e28e179-25f6-4bf5-b769-e3c30da4480a-plugins-config-volume\") pod \"6e28e179-25f6-4bf5-b769-e3c30da4480a\" (UID: \"6e28e179-25f6-4bf5-b769-e3c30da4480a\") "
Sep 07 21:18:51.936318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936181    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h" event={"ID":"dff48924-2038-4fbc-80cc-8e29f755b98b","Type":"ContainerDied","Data":"974310f274c84a24f1832ade6f6be2356edec29e6ac7ed8ac3851c120a3d0d84"}
Sep 07 21:18:51.936318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936215    2565 scope.go:117] "RemoveContainer" containerID="8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9"
Sep 07 21:18:51.936318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936271    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4p794\" (UniqueName: \"kubernetes.io/projected/dff48924-2038-4fbc-80cc-8e29f755b98b-kube-api-access-4p794\") pod \"dff48924-2038-4fbc-80cc-8e29f755b98b\" (UID: \"dff48924-2038-4fbc-80cc-8e29f755b98b\") "
Sep 07 21:18:51.936318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936319    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff48924-2038-4fbc-80cc-8e29f755b98b-plugins-config-volume\") pod \"dff48924-2038-4fbc-80cc-8e29f755b98b\" (UID: \"dff48924-2038-4fbc-80cc-8e29f755b98b\") "
Sep 07 21:18:51.936661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936555    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e28e179-25f6-4bf5-b769-e3c30da4480a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6e28e179-25f6-4bf5-b769-e3c30da4480a" (UID: "6e28e179-25f6-4bf5-b769-e3c30da4480a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:18:51.936661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936644    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e28e179-25f6-4bf5-b769-e3c30da4480a-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:18:51.936916 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.936879    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff48924-2038-4fbc-80cc-8e29f755b98b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dff48924-2038-4fbc-80cc-8e29f755b98b" (UID: "dff48924-2038-4fbc-80cc-8e29f755b98b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:18:51.937690 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.937663    2565 generic.go:356] "Generic (PLEG): container finished" podID="6e28e179-25f6-4bf5-b769-e3c30da4480a" containerID="08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02" exitCode=0
Sep 07 21:18:51.937824 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.937715    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"
Sep 07 21:18:51.937824 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.937765    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm" event={"ID":"6e28e179-25f6-4bf5-b769-e3c30da4480a","Type":"ContainerDied","Data":"08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02"}
Sep 07 21:18:51.937950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.937834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm" event={"ID":"6e28e179-25f6-4bf5-b769-e3c30da4480a","Type":"ContainerDied","Data":"ccddad589eb271b516c3b0a769913d17c44d5386ec39882c6a15ad1f5c05c995"}
Sep 07 21:18:51.938515 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.938491    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff48924-2038-4fbc-80cc-8e29f755b98b-kube-api-access-4p794" (OuterVolumeSpecName: "kube-api-access-4p794") pod "dff48924-2038-4fbc-80cc-8e29f755b98b" (UID: "dff48924-2038-4fbc-80cc-8e29f755b98b"). InnerVolumeSpecName "kube-api-access-4p794". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:18:51.938637 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.938614    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e28e179-25f6-4bf5-b769-e3c30da4480a-kube-api-access-qqtlh" (OuterVolumeSpecName: "kube-api-access-qqtlh") pod "6e28e179-25f6-4bf5-b769-e3c30da4480a" (UID: "6e28e179-25f6-4bf5-b769-e3c30da4480a"). InnerVolumeSpecName "kube-api-access-qqtlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:18:51.945003 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.944986    2565 scope.go:117] "RemoveContainer" containerID="8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9"
Sep 07 21:18:51.945260 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:18:51.945242    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9\": container with ID starting with 8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9 not found: ID does not exist" containerID="8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9"
Sep 07 21:18:51.945310 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.945268    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9"} err="failed to get container status \"8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9\": rpc error: code = NotFound desc = could not find container \"8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9\": container with ID starting with 8d3454cfa0fbcda55b390cef91cfcc7abb2aaf132e7a43139f2591290e5fd5b9 not found: ID does not exist"
Sep 07 21:18:51.945310 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.945289    2565 scope.go:117] "RemoveContainer" containerID="08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02"
Sep 07 21:18:51.952735 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.952718    2565 scope.go:117] "RemoveContainer" containerID="08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02"
Sep 07 21:18:51.953001 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:18:51.952981    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02\": container with ID starting with 08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02 not found: ID does not exist" containerID="08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02"
Sep 07 21:18:51.953053 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:51.953006    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02"} err="failed to get container status \"08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02\": rpc error: code = NotFound desc = could not find container \"08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02\": container with ID starting with 08f00c776159336b8810edf7b2bc131b3bd77876ed8bcc0573943a3d4a77ea02 not found: ID does not exist"
Sep 07 21:18:52.037327 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.037299    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqtlh\" (UniqueName: \"kubernetes.io/projected/6e28e179-25f6-4bf5-b769-e3c30da4480a-kube-api-access-qqtlh\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:18:52.037327 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.037326    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p794\" (UniqueName: \"kubernetes.io/projected/dff48924-2038-4fbc-80cc-8e29f755b98b-kube-api-access-4p794\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:18:52.037507 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.037336    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dff48924-2038-4fbc-80cc-8e29f755b98b-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:18:52.257857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.257830    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"]
Sep 07 21:18:52.261053 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.261028    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f447ae-64bbc4b86f-rcs8h"]
Sep 07 21:18:52.271867 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.271847    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"]
Sep 07 21:18:52.277368 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.277348    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f447ae-7c765d6cf4-pg2dm"]
Sep 07 21:18:52.363762 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.363719    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3846750b-c3e5-47ab-86bd-7a32c1111756" path="/var/lib/kubelet/pods/3846750b-c3e5-47ab-86bd-7a32c1111756/volumes"
Sep 07 21:18:52.364171 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.364155    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e28e179-25f6-4bf5-b769-e3c30da4480a" path="/var/lib/kubelet/pods/6e28e179-25f6-4bf5-b769-e3c30da4480a/volumes"
Sep 07 21:18:52.364475 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:18:52.364464    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff48924-2038-4fbc-80cc-8e29f755b98b" path="/var/lib/kubelet/pods/dff48924-2038-4fbc-80cc-8e29f755b98b/volumes"
Sep 07 21:19:02.052231 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:02.052194    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:19:03.217047 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:03.217014    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"]
Sep 07 21:19:03.217535 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:03.217250    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy" containerID="cri-o://777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3" gracePeriod=30
Sep 07 21:19:06.128033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:06.127996    2565 patch_prober.go:28] interesting pod/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 21:19:06.128404 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:06.128070    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 21:19:08.471224 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.471199    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:19:08.585112 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585089    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-token\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585244 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585134    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-credential-socket\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585244 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585175    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-certs\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585403 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585371    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgcds\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-kube-api-access-kgcds\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585488 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585436    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:19:08.585488 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585449    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:19:08.585488 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585479    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ce3ecc-7014-4e36-884f-6dca0251343f-istiod-ca-cert\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585654 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585507    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-socket\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585654 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585534    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-envoy\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585654 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585562    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-data\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585778 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585729    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-podinfo\") pod \"68ce3ecc-7014-4e36-884f-6dca0251343f\" (UID: \"68ce3ecc-7014-4e36-884f-6dca0251343f\") "
Sep 07 21:19:08.585778 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585761    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:19:08.585887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585843    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ce3ecc-7014-4e36-884f-6dca0251343f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:19:08.585952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.585898    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-data" (OuterVolumeSpecName: "istio-data") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:19:08.586145 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.586121    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-credential-socket\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:08.586229 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.586150    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:08.586229 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.586166    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68ce3ecc-7014-4e36-884f-6dca0251343f-istiod-ca-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:08.586229 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.586179    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-workload-socket\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:08.586229 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.586191    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-data\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:08.587462 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.587439    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-token" (OuterVolumeSpecName: "istio-token") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:19:08.587637 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.587620    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:19:08.587691 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.587657    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-kube-api-access-kgcds" (OuterVolumeSpecName: "kube-api-access-kgcds") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "kube-api-access-kgcds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:19:08.587733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.587709    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "68ce3ecc-7014-4e36-884f-6dca0251343f" (UID: "68ce3ecc-7014-4e36-884f-6dca0251343f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 21:19:08.687183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.687152    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-podinfo\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:08.687183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.687180    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-token\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:08.687183 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.687189    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcds\" (UniqueName: \"kubernetes.io/projected/68ce3ecc-7014-4e36-884f-6dca0251343f-kube-api-access-kgcds\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:08.687567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:08.687197    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68ce3ecc-7014-4e36-884f-6dca0251343f-istio-envoy\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:09.028911 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.028825    2565 generic.go:356] "Generic (PLEG): container finished" podID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerID="777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3" exitCode=0
Sep 07 21:19:09.028911 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.028862    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" event={"ID":"68ce3ecc-7014-4e36-884f-6dca0251343f","Type":"ContainerDied","Data":"777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3"}
Sep 07 21:19:09.028911 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.028875    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"
Sep 07 21:19:09.028911 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.028900    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2" event={"ID":"68ce3ecc-7014-4e36-884f-6dca0251343f","Type":"ContainerDied","Data":"757a70643213c164907cd7d2cace35baf318ab8ce1ebf51acfd50b939a545098"}
Sep 07 21:19:09.029208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.028921    2565 scope.go:117] "RemoveContainer" containerID="777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3"
Sep 07 21:19:09.037852 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.037836    2565 scope.go:117] "RemoveContainer" containerID="66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79"
Sep 07 21:19:09.045717 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.045702    2565 scope.go:117] "RemoveContainer" containerID="777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3"
Sep 07 21:19:09.045977 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:19:09.045941    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3\": container with ID starting with 777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3 not found: ID does not exist" containerID="777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3"
Sep 07 21:19:09.046028 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.045983    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3"} err="failed to get container status \"777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3\": rpc error: code = NotFound desc = could not find container \"777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3\": container with ID starting with 777abd34cf6201eee37deffac715f78aa21d109a320c7e4d8e205fe0cbe750a3 not found: ID does not exist"
Sep 07 21:19:09.046028 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.046002    2565 scope.go:117] "RemoveContainer" containerID="66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79"
Sep 07 21:19:09.046236 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:19:09.046222    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79\": container with ID starting with 66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79 not found: ID does not exist" containerID="66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79"
Sep 07 21:19:09.046276 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.046240    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79"} err="failed to get container status \"66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79\": rpc error: code = NotFound desc = could not find container \"66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79\": container with ID starting with 66caf1e9a944ee80047cb180c779eec3c9a8ee3a823dc4998071c23dbc6d1a79 not found: ID does not exist"
Sep 07 21:19:09.050092 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.050069    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"]
Sep 07 21:19:09.053938 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:09.053916    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-f447ae-openshift-default-6b79fb8c88-47wz2"]
Sep 07 21:19:10.363733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:10.363699    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" path="/var/lib/kubelet/pods/68ce3ecc-7014-4e36-884f-6dca0251343f/volumes"
Sep 07 21:19:13.337727 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.337693    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"]
Sep 07 21:19:13.338206 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338185    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e28e179-25f6-4bf5-b769-e3c30da4480a" containerName="payload-pre-processing"
Sep 07 21:19:13.338254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338211    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e28e179-25f6-4bf5-b769-e3c30da4480a" containerName="payload-pre-processing"
Sep 07 21:19:13.338254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338226    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy"
Sep 07 21:19:13.338254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338236    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy"
Sep 07 21:19:13.338254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338251    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy"
Sep 07 21:19:13.338381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338260    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy"
Sep 07 21:19:13.338381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338277    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff48924-2038-4fbc-80cc-8e29f755b98b" containerName="payload-processing"
Sep 07 21:19:13.338381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338286    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff48924-2038-4fbc-80cc-8e29f755b98b" containerName="payload-processing"
Sep 07 21:19:13.338381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338316    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3846750b-c3e5-47ab-86bd-7a32c1111756" containerName="revoke-keys"
Sep 07 21:19:13.338381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338324    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="3846750b-c3e5-47ab-86bd-7a32c1111756" containerName="revoke-keys"
Sep 07 21:19:13.338593 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338398    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="3846750b-c3e5-47ab-86bd-7a32c1111756" containerName="revoke-keys"
Sep 07 21:19:13.338593 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338408    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e28e179-25f6-4bf5-b769-e3c30da4480a" containerName="payload-pre-processing"
Sep 07 21:19:13.338593 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338413    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy"
Sep 07 21:19:13.338593 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338420    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff48924-2038-4fbc-80cc-8e29f755b98b" containerName="payload-processing"
Sep 07 21:19:13.338593 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.338426    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ce3ecc-7014-4e36-884f-6dca0251343f" containerName="istio-proxy"
Sep 07 21:19:13.348715 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.348684    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.351561 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.351537    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6223ec\"/\"kube-root-ca.crt\""
Sep 07 21:19:13.351720 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.351698    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6223ec\"/\"noten-model-6223ec-kserve-self-signed-certs\""
Sep 07 21:19:13.352441 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.352270    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-6223ec\"/\"openshift-service-ca.crt\""
Sep 07 21:19:13.352441 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.352362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-6223ec\"/\"default-dockercfg-m6nzk\""
Sep 07 21:19:13.353231 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.353201    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"]
Sep 07 21:19:13.428783 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.428757    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbbf1b6-5158-4780-a968-758132cdeb21-tls-certs\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.428901 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.428792    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-home\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.428901 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.428830    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-tmp-dir\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.428901 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.428847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-dshm\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.428901 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.428883    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx7zd\" (UniqueName: \"kubernetes.io/projected/fcbbf1b6-5158-4780-a968-758132cdeb21-kube-api-access-xx7zd\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.429061 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.428919    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-model-cache\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.529848 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.529808    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx7zd\" (UniqueName: \"kubernetes.io/projected/fcbbf1b6-5158-4780-a968-758132cdeb21-kube-api-access-xx7zd\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.529848 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.529851    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-model-cache\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.530116 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.529892    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbbf1b6-5158-4780-a968-758132cdeb21-tls-certs\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.530116 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.529915    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-home\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.530116 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.529958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-tmp-dir\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.530116 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.530014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-dshm\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.530388 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.530363    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-home\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.530474 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.530372    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-model-cache\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.530474 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.530443    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-tmp-dir\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.532365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.532341    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-dshm\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.532596 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.532581    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbbf1b6-5158-4780-a968-758132cdeb21-tls-certs\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.537621 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.537601    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx7zd\" (UniqueName: \"kubernetes.io/projected/fcbbf1b6-5158-4780-a968-758132cdeb21-kube-api-access-xx7zd\") pod \"noten-model-6223ec-kserve-6f5995576c-vhhhs\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") " pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.660409 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.660316    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:13.932583 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:13.932559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"]
Sep 07 21:19:13.934514 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:19:13.934485    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfcbbf1b6_5158_4780_a968_758132cdeb21.slice/crio-f6ef6dae582f725c6588816970d9d58fc751bf8dd83c35759396327eecf3c504 WatchSource:0}: Error finding container f6ef6dae582f725c6588816970d9d58fc751bf8dd83c35759396327eecf3c504: Status 404 returned error can't find the container with id f6ef6dae582f725c6588816970d9d58fc751bf8dd83c35759396327eecf3c504
Sep 07 21:19:14.056393 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:14.056357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs" event={"ID":"fcbbf1b6-5158-4780-a968-758132cdeb21","Type":"ContainerStarted","Data":"ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576"}
Sep 07 21:19:14.056556 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:14.056404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs" event={"ID":"fcbbf1b6-5158-4780-a968-758132cdeb21","Type":"ContainerStarted","Data":"f6ef6dae582f725c6588816970d9d58fc751bf8dd83c35759396327eecf3c504"}
Sep 07 21:19:14.072205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:14.072156    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs" podStartSLOduration=1.072139399 podStartE2EDuration="1.072139399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:19:14.071174113 +0000 UTC m=+1046.152701456" watchObservedRunningTime="2026-09-07 21:19:14.072139399 +0000 UTC m=+1046.153666738"
Sep 07 21:19:23.653111 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:23.653074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:19:23.660530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:23.660507    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:23.660653 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:23.660546    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:23.673066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:23.673039    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:24.114619 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:24.114589    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:19:34.532842 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:34.532808    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:19:35.125397 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.125365    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"]
Sep 07 21:19:35.128816 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.128799    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:35.131763 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.131738    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f838f49c-dockercfg-jr9r6\""
Sep 07 21:19:35.131902 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.131807    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f838f49c\""
Sep 07 21:19:35.140310 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.140288    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"]
Sep 07 21:19:35.223713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.223672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-plugins-config-volume\") pod \"payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml\" (UID: \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\") " pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:35.224023 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.223868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw57w\" (UniqueName: \"kubernetes.io/projected/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-kube-api-access-fw57w\") pod \"payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml\" (UID: \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\") " pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:35.325303 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.325268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw57w\" (UniqueName: \"kubernetes.io/projected/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-kube-api-access-fw57w\") pod \"payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml\" (UID: \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\") " pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:35.325489 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.325342    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-plugins-config-volume\") pod \"payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml\" (UID: \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\") " pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:35.326087 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.326070    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-plugins-config-volume\") pod \"payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml\" (UID: \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\") " pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:35.333529 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.333511    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw57w\" (UniqueName: \"kubernetes.io/projected/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-kube-api-access-fw57w\") pod \"payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml\" (UID: \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\") " pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:35.441055 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.440980    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:35.711315 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:35.711284    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"]
Sep 07 21:19:35.713903 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:19:35.713875    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7b273ec_9f6e_47e0_8f40_5f6fbe703996.slice/crio-068b13ccb52e0d5f5ee0c96fc8b958c5979a86b9a12f6187d4065f2dbfda4dc3 WatchSource:0}: Error finding container 068b13ccb52e0d5f5ee0c96fc8b958c5979a86b9a12f6187d4065f2dbfda4dc3: Status 404 returned error can't find the container with id 068b13ccb52e0d5f5ee0c96fc8b958c5979a86b9a12f6187d4065f2dbfda4dc3
Sep 07 21:19:36.165283 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:36.165248    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml" event={"ID":"c7b273ec-9f6e-47e0-8f40-5f6fbe703996","Type":"ContainerStarted","Data":"242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be"}
Sep 07 21:19:36.165499 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:36.165311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml" event={"ID":"c7b273ec-9f6e-47e0-8f40-5f6fbe703996","Type":"ContainerStarted","Data":"068b13ccb52e0d5f5ee0c96fc8b958c5979a86b9a12f6187d4065f2dbfda4dc3"}
Sep 07 21:19:36.165499 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:36.165345    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:36.182217 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:36.182170    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml" podStartSLOduration=1.182155149 podStartE2EDuration="1.182155149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:19:36.180553489 +0000 UTC m=+1068.262080831" watchObservedRunningTime="2026-09-07 21:19:36.182155149 +0000 UTC m=+1068.263682490"
Sep 07 21:19:47.172892 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:47.172860    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:19:49.666984 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:49.666824    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"]
Sep 07 21:19:50.840165 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:50.840131    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"]
Sep 07 21:19:51.639761 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.639723    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"]
Sep 07 21:19:51.765672 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.765635    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"]
Sep 07 21:19:51.770207 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.770188    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"
Sep 07 21:19:51.772540 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.772516    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:19:51.772643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.772545    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:19:51.777863 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.777814    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"]
Sep 07 21:19:51.870072 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.870035    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e037af6-7064-4324-b688-58973c492608-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr\" (UID: \"8e037af6-7064-4324-b688-58973c492608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"
Sep 07 21:19:51.971503 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.971416    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e037af6-7064-4324-b688-58973c492608-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr\" (UID: \"8e037af6-7064-4324-b688-58973c492608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"
Sep 07 21:19:51.972040 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:51.972019    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e037af6-7064-4324-b688-58973c492608-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr\" (UID: \"8e037af6-7064-4324-b688-58973c492608\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"
Sep 07 21:19:52.081344 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:52.081315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"
Sep 07 21:19:52.349642 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:52.349620    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"]
Sep 07 21:19:52.351823 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:19:52.351790    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e037af6_7064_4324_b688_58973c492608.slice/crio-377f94e3ae697294f26d7acddeaead3d8f785459e947e961ff08e65b91626e27 WatchSource:0}: Error finding container 377f94e3ae697294f26d7acddeaead3d8f785459e947e961ff08e65b91626e27: Status 404 returned error can't find the container with id 377f94e3ae697294f26d7acddeaead3d8f785459e947e961ff08e65b91626e27
Sep 07 21:19:52.749418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:52.749388    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:19:53.258689 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:53.258648    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e037af6-7064-4324-b688-58973c492608" containerID="a5dace73ca8c8da7c7a0fda1b755d420571469e65ef9b2e113c62084665c0973" exitCode=0
Sep 07 21:19:53.259066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:53.258743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr" event={"ID":"8e037af6-7064-4324-b688-58973c492608","Type":"ContainerDied","Data":"a5dace73ca8c8da7c7a0fda1b755d420571469e65ef9b2e113c62084665c0973"}
Sep 07 21:19:53.259066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:53.258828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr" event={"ID":"8e037af6-7064-4324-b688-58973c492608","Type":"ContainerStarted","Data":"377f94e3ae697294f26d7acddeaead3d8f785459e947e961ff08e65b91626e27"}
Sep 07 21:19:54.406901 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:54.406878    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"
Sep 07 21:19:54.494529 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:54.494496    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e037af6-7064-4324-b688-58973c492608-maas-api-service-ca\") pod \"8e037af6-7064-4324-b688-58973c492608\" (UID: \"8e037af6-7064-4324-b688-58973c492608\") "
Sep 07 21:19:54.494834 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:54.494813    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e037af6-7064-4324-b688-58973c492608-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8e037af6-7064-4324-b688-58973c492608" (UID: "8e037af6-7064-4324-b688-58973c492608"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:19:54.595750 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:54.595717    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e037af6-7064-4324-b688-58973c492608-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:19:55.272514 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:55.272483    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"
Sep 07 21:19:55.272687 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:55.272483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr" event={"ID":"8e037af6-7064-4324-b688-58973c492608","Type":"ContainerDied","Data":"377f94e3ae697294f26d7acddeaead3d8f785459e947e961ff08e65b91626e27"}
Sep 07 21:19:55.272687 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:19:55.272607    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="377f94e3ae697294f26d7acddeaead3d8f785459e947e961ff08e65b91626e27"
Sep 07 21:20:02.861854 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:02.861814    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"]
Sep 07 21:20:02.863879 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:02.863855    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-06f0f6-gl9xr"]
Sep 07 21:20:04.362971 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:04.362924    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e037af6-7064-4324-b688-58973c492608" path="/var/lib/kubelet/pods/8e037af6-7064-4324-b688-58973c492608/volumes"
Sep 07 21:20:04.687369 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:04.687277    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x" podUID="b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" containerName="main" containerID="cri-o://322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430" gracePeriod=45
Sep 07 21:20:04.940518 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:04.940463    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:20:05.082722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.082687    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-model-cache\") pod \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") "
Sep 07 21:20:05.082722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.082726    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b5cxx\" (UniqueName: \"kubernetes.io/projected/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-kube-api-access-b5cxx\") pod \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") "
Sep 07 21:20:05.083006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.082752    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tls-certs\") pod \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") "
Sep 07 21:20:05.083006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.082800    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-dshm\") pod \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") "
Sep 07 21:20:05.083006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.082838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-home\") pod \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") "
Sep 07 21:20:05.083006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.082919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tmp-dir\") pod \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\" (UID: \"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc\") "
Sep 07 21:20:05.083213 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.083069    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-model-cache" (OuterVolumeSpecName: "model-cache") pod "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" (UID: "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:05.083265 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.083226    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-home" (OuterVolumeSpecName: "home") pod "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" (UID: "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:05.083265 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.083236    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" (UID: "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:05.083424 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.083344    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:05.083424 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.083358    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:05.083424 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.083367    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:05.084996 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.084958    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-dshm" (OuterVolumeSpecName: "dshm") pod "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" (UID: "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:05.085103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.085084    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" (UID: "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:20:05.085143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.085097    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-kube-api-access-b5cxx" (OuterVolumeSpecName: "kube-api-access-b5cxx") pod "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" (UID: "b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc"). InnerVolumeSpecName "kube-api-access-b5cxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:20:05.184721 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.184691    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5cxx\" (UniqueName: \"kubernetes.io/projected/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-kube-api-access-b5cxx\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:05.184721 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.184718    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:05.184891 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.184730    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:05.324169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.324080    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" containerID="322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430" exitCode=0
Sep 07 21:20:05.324169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.324123    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"
Sep 07 21:20:05.324169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.324143    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x" event={"ID":"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc","Type":"ContainerDied","Data":"322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430"}
Sep 07 21:20:05.324447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.324175    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x" event={"ID":"b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc","Type":"ContainerDied","Data":"478f3ce65ec9e828ac3af0426cb348f76f3c2eedfb88a5891b73079f75e84774"}
Sep 07 21:20:05.324447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.324192    2565 scope.go:117] "RemoveContainer" containerID="322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430"
Sep 07 21:20:05.333308 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.333291    2565 scope.go:117] "RemoveContainer" containerID="322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430"
Sep 07 21:20:05.333578 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:20:05.333561    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430\": container with ID starting with 322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430 not found: ID does not exist" containerID="322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430"
Sep 07 21:20:05.333626 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.333587    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430"} err="failed to get container status \"322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430\": rpc error: code = NotFound desc = could not find container \"322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430\": container with ID starting with 322a1499f42816e93a93ad96ec648a3c341d324652c6005b943b6c640a082430 not found: ID does not exist"
Sep 07 21:20:05.344002 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.343979    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"]
Sep 07 21:20:05.349696 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.349676    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-06f0f6/test-model-06f0f6-kserve-76c4cd6997-4vv7x"]
Sep 07 21:20:05.860077 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:05.860034    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs" podUID="fcbbf1b6-5158-4780-a968-758132cdeb21" containerName="main" containerID="cri-o://ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576" gracePeriod=45
Sep 07 21:20:06.110924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.110862    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:20:06.194502 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.194467    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbbf1b6-5158-4780-a968-758132cdeb21-tls-certs\") pod \"fcbbf1b6-5158-4780-a968-758132cdeb21\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") "
Sep 07 21:20:06.194695 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.194514    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-model-cache\") pod \"fcbbf1b6-5158-4780-a968-758132cdeb21\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") "
Sep 07 21:20:06.194695 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.194553    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx7zd\" (UniqueName: \"kubernetes.io/projected/fcbbf1b6-5158-4780-a968-758132cdeb21-kube-api-access-xx7zd\") pod \"fcbbf1b6-5158-4780-a968-758132cdeb21\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") "
Sep 07 21:20:06.194695 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.194631    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-home\") pod \"fcbbf1b6-5158-4780-a968-758132cdeb21\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") "
Sep 07 21:20:06.194872 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.194737    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-dshm\") pod \"fcbbf1b6-5158-4780-a968-758132cdeb21\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") "
Sep 07 21:20:06.194872 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.194767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-tmp-dir\") pod \"fcbbf1b6-5158-4780-a968-758132cdeb21\" (UID: \"fcbbf1b6-5158-4780-a968-758132cdeb21\") "
Sep 07 21:20:06.194872 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.194853    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-home" (OuterVolumeSpecName: "home") pod "fcbbf1b6-5158-4780-a968-758132cdeb21" (UID: "fcbbf1b6-5158-4780-a968-758132cdeb21"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:06.195065 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.194859    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-model-cache" (OuterVolumeSpecName: "model-cache") pod "fcbbf1b6-5158-4780-a968-758132cdeb21" (UID: "fcbbf1b6-5158-4780-a968-758132cdeb21"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:06.195141 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.195116    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:06.195190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.195151    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:06.195190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.195114    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fcbbf1b6-5158-4780-a968-758132cdeb21" (UID: "fcbbf1b6-5158-4780-a968-758132cdeb21"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:06.196867 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.196839    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcbbf1b6-5158-4780-a968-758132cdeb21-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fcbbf1b6-5158-4780-a968-758132cdeb21" (UID: "fcbbf1b6-5158-4780-a968-758132cdeb21"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:20:06.196867 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.196855    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbbf1b6-5158-4780-a968-758132cdeb21-kube-api-access-xx7zd" (OuterVolumeSpecName: "kube-api-access-xx7zd") pod "fcbbf1b6-5158-4780-a968-758132cdeb21" (UID: "fcbbf1b6-5158-4780-a968-758132cdeb21"). InnerVolumeSpecName "kube-api-access-xx7zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:20:06.197038 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.196906    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-dshm" (OuterVolumeSpecName: "dshm") pod "fcbbf1b6-5158-4780-a968-758132cdeb21" (UID: "fcbbf1b6-5158-4780-a968-758132cdeb21"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:06.296379 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.296332    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcbbf1b6-5158-4780-a968-758132cdeb21-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:06.296379 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.296375    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx7zd\" (UniqueName: \"kubernetes.io/projected/fcbbf1b6-5158-4780-a968-758132cdeb21-kube-api-access-xx7zd\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:06.296379 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.296388    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:06.296593 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.296400    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcbbf1b6-5158-4780-a968-758132cdeb21-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:06.330006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.329947    2565 generic.go:356] "Generic (PLEG): container finished" podID="fcbbf1b6-5158-4780-a968-758132cdeb21" containerID="ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576" exitCode=0
Sep 07 21:20:06.330140 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.330012    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"
Sep 07 21:20:06.330140 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.330038    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs" event={"ID":"fcbbf1b6-5158-4780-a968-758132cdeb21","Type":"ContainerDied","Data":"ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576"}
Sep 07 21:20:06.330140 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.330080    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs" event={"ID":"fcbbf1b6-5158-4780-a968-758132cdeb21","Type":"ContainerDied","Data":"f6ef6dae582f725c6588816970d9d58fc751bf8dd83c35759396327eecf3c504"}
Sep 07 21:20:06.330140 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.330104    2565 scope.go:117] "RemoveContainer" containerID="ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576"
Sep 07 21:20:06.339576 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.339557    2565 scope.go:117] "RemoveContainer" containerID="ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576"
Sep 07 21:20:06.339827 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:20:06.339808    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576\": container with ID starting with ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576 not found: ID does not exist" containerID="ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576"
Sep 07 21:20:06.339887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.339836    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576"} err="failed to get container status \"ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576\": rpc error: code = NotFound desc = could not find container \"ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576\": container with ID starting with ef64e7d205daf2d33133f1882903f675f0357733b20bf0de14f03557508ad576 not found: ID does not exist"
Sep 07 21:20:06.352810 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.352790    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"]
Sep 07 21:20:06.356877 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.356856    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-6223ec/noten-model-6223ec-kserve-6f5995576c-vhhhs"]
Sep 07 21:20:06.364392 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.364340    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" path="/var/lib/kubelet/pods/b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc/volumes"
Sep 07 21:20:06.364786 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.364772    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcbbf1b6-5158-4780-a968-758132cdeb21" path="/var/lib/kubelet/pods/fcbbf1b6-5158-4780-a968-758132cdeb21/volumes"
Sep 07 21:20:06.659153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.659059    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv" podUID="c9998df7-d3ec-4cad-8d61-e9be1abde86b" containerName="main" containerID="cri-o://38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736" gracePeriod=45
Sep 07 21:20:06.905460 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.905438    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:20:06.959658 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:06.959580    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:20:07.003016 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.002990    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-model-cache\") pod \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") "
Sep 07 21:20:07.003158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003033    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tls-certs\") pod \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") "
Sep 07 21:20:07.003158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003056    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tmp-dir\") pod \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") "
Sep 07 21:20:07.003158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003073    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-dshm\") pod \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") "
Sep 07 21:20:07.003158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m972c\" (UniqueName: \"kubernetes.io/projected/c9998df7-d3ec-4cad-8d61-e9be1abde86b-kube-api-access-m972c\") pod \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") "
Sep 07 21:20:07.003376 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003180    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-home\") pod \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\" (UID: \"c9998df7-d3ec-4cad-8d61-e9be1abde86b\") "
Sep 07 21:20:07.003376 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003274    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9998df7-d3ec-4cad-8d61-e9be1abde86b" (UID: "c9998df7-d3ec-4cad-8d61-e9be1abde86b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:07.003376 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003309    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9998df7-d3ec-4cad-8d61-e9be1abde86b" (UID: "c9998df7-d3ec-4cad-8d61-e9be1abde86b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:07.003510 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003431    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:07.003510 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003451    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:07.003510 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.003489    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-home" (OuterVolumeSpecName: "home") pod "c9998df7-d3ec-4cad-8d61-e9be1abde86b" (UID: "c9998df7-d3ec-4cad-8d61-e9be1abde86b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:07.005307 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.005276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-dshm" (OuterVolumeSpecName: "dshm") pod "c9998df7-d3ec-4cad-8d61-e9be1abde86b" (UID: "c9998df7-d3ec-4cad-8d61-e9be1abde86b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:07.005422 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.005306    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9998df7-d3ec-4cad-8d61-e9be1abde86b" (UID: "c9998df7-d3ec-4cad-8d61-e9be1abde86b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:20:07.005422 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.005368    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9998df7-d3ec-4cad-8d61-e9be1abde86b-kube-api-access-m972c" (OuterVolumeSpecName: "kube-api-access-m972c") pod "c9998df7-d3ec-4cad-8d61-e9be1abde86b" (UID: "c9998df7-d3ec-4cad-8d61-e9be1abde86b"). InnerVolumeSpecName "kube-api-access-m972c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:20:07.104343 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.104306    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:07.104343 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.104337    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9998df7-d3ec-4cad-8d61-e9be1abde86b-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:07.104343 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.104349    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9998df7-d3ec-4cad-8d61-e9be1abde86b-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:07.104647 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.104361    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m972c\" (UniqueName: \"kubernetes.io/projected/c9998df7-d3ec-4cad-8d61-e9be1abde86b-kube-api-access-m972c\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:07.336731 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.336699    2565 generic.go:356] "Generic (PLEG): container finished" podID="c9998df7-d3ec-4cad-8d61-e9be1abde86b" containerID="38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736" exitCode=0
Sep 07 21:20:07.336918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.336744    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"
Sep 07 21:20:07.336918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.336790    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv" event={"ID":"c9998df7-d3ec-4cad-8d61-e9be1abde86b","Type":"ContainerDied","Data":"38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736"}
Sep 07 21:20:07.336918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.336834    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv" event={"ID":"c9998df7-d3ec-4cad-8d61-e9be1abde86b","Type":"ContainerDied","Data":"6abcbd1f84b43251cd04996936567218b8506049169dffe4d360a3dfc5ec3c85"}
Sep 07 21:20:07.336918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.336852    2565 scope.go:117] "RemoveContainer" containerID="38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736"
Sep 07 21:20:07.345463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.345445    2565 scope.go:117] "RemoveContainer" containerID="38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736"
Sep 07 21:20:07.345728 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:20:07.345710    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736\": container with ID starting with 38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736 not found: ID does not exist" containerID="38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736"
Sep 07 21:20:07.345775 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.345736    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736"} err="failed to get container status \"38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736\": rpc error: code = NotFound desc = could not find container \"38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736\": container with ID starting with 38824300cad27fdb3be9f49c5941d877829af3f26273988332cbb1ecd2d47736 not found: ID does not exist"
Sep 07 21:20:07.356560 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.356539    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"]
Sep 07 21:20:07.360225 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:07.360199    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3102b2/test-model-3102b2-kserve-67556d4f49-rpksv"]
Sep 07 21:20:08.362804 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:08.362760    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9998df7-d3ec-4cad-8d61-e9be1abde86b" path="/var/lib/kubelet/pods/c9998df7-d3ec-4cad-8d61-e9be1abde86b/volumes"
Sep 07 21:20:15.914931 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:15.914888    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"]
Sep 07 21:20:15.915433 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:15.915174    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml" podUID="c7b273ec-9f6e-47e0-8f40-5f6fbe703996" containerName="payload-processing" containerID="cri-o://242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be" gracePeriod=30
Sep 07 21:20:16.160331 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.160307    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:20:16.287235 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.287155    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw57w\" (UniqueName: \"kubernetes.io/projected/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-kube-api-access-fw57w\") pod \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\" (UID: \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\") "
Sep 07 21:20:16.287235 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.287234    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-plugins-config-volume\") pod \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\" (UID: \"c7b273ec-9f6e-47e0-8f40-5f6fbe703996\") "
Sep 07 21:20:16.287605 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.287579    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7b273ec-9f6e-47e0-8f40-5f6fbe703996" (UID: "c7b273ec-9f6e-47e0-8f40-5f6fbe703996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:20:16.289450 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.289404    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-kube-api-access-fw57w" (OuterVolumeSpecName: "kube-api-access-fw57w") pod "c7b273ec-9f6e-47e0-8f40-5f6fbe703996" (UID: "c7b273ec-9f6e-47e0-8f40-5f6fbe703996"). InnerVolumeSpecName "kube-api-access-fw57w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:20:16.388208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.388158    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:16.388208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.388187    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw57w\" (UniqueName: \"kubernetes.io/projected/c7b273ec-9f6e-47e0-8f40-5f6fbe703996-kube-api-access-fw57w\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:16.392245 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.392219    2565 generic.go:356] "Generic (PLEG): container finished" podID="c7b273ec-9f6e-47e0-8f40-5f6fbe703996" containerID="242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be" exitCode=0
Sep 07 21:20:16.392347 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.392254    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml" event={"ID":"c7b273ec-9f6e-47e0-8f40-5f6fbe703996","Type":"ContainerDied","Data":"242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be"}
Sep 07 21:20:16.392347 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.392266    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"
Sep 07 21:20:16.392347 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.392289    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml" event={"ID":"c7b273ec-9f6e-47e0-8f40-5f6fbe703996","Type":"ContainerDied","Data":"068b13ccb52e0d5f5ee0c96fc8b958c5979a86b9a12f6187d4065f2dbfda4dc3"}
Sep 07 21:20:16.392347 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.392310    2565 scope.go:117] "RemoveContainer" containerID="242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be"
Sep 07 21:20:16.400616 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.400597    2565 scope.go:117] "RemoveContainer" containerID="242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be"
Sep 07 21:20:16.400875 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:20:16.400858    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be\": container with ID starting with 242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be not found: ID does not exist" containerID="242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be"
Sep 07 21:20:16.400920 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.400884    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be"} err="failed to get container status \"242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be\": rpc error: code = NotFound desc = could not find container \"242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be\": container with ID starting with 242f451baf9b1e57c953af364d2bd5efcb73c22e2e8a58027d465cf552ec30be not found: ID does not exist"
Sep 07 21:20:16.407011 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.406990    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"]
Sep 07 21:20:16.410857 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:16.410838    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f838f49c-85f7497965-9s5ml"]
Sep 07 21:20:18.363461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:18.363429    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7b273ec-9f6e-47e0-8f40-5f6fbe703996" path="/var/lib/kubelet/pods/c7b273ec-9f6e-47e0-8f40-5f6fbe703996/volumes"
Sep 07 21:20:18.659670 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:18.659588    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:20:26.749631 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:26.749594    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"]
Sep 07 21:20:26.750043 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:26.749823    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy" containerID="cri-o://7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2" gracePeriod=30
Sep 07 21:20:29.964410 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:29.964378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:20:29.974836 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:29.974799    2565 patch_prober.go:28] interesting pod/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 21:20:29.974986 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:29.974858    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 21:20:31.942861 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.942830    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"]
Sep 07 21:20:31.943339 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943319    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e037af6-7064-4324-b688-58973c492608" containerName="revoke-keys"
Sep 07 21:20:31.943415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943342    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e037af6-7064-4324-b688-58973c492608" containerName="revoke-keys"
Sep 07 21:20:31.943415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943378    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" containerName="main"
Sep 07 21:20:31.943415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943388    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" containerName="main"
Sep 07 21:20:31.943415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943397    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcbbf1b6-5158-4780-a968-758132cdeb21" containerName="main"
Sep 07 21:20:31.943415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943406    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcbbf1b6-5158-4780-a968-758132cdeb21" containerName="main"
Sep 07 21:20:31.943415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943415    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9998df7-d3ec-4cad-8d61-e9be1abde86b" containerName="main"
Sep 07 21:20:31.943713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943424    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9998df7-d3ec-4cad-8d61-e9be1abde86b" containerName="main"
Sep 07 21:20:31.943713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943447    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7b273ec-9f6e-47e0-8f40-5f6fbe703996" containerName="payload-processing"
Sep 07 21:20:31.943713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943455    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7b273ec-9f6e-47e0-8f40-5f6fbe703996" containerName="payload-processing"
Sep 07 21:20:31.943713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943542    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2d7fb9b-bd1a-41bc-bd1d-9ffccfd9b8cc" containerName="main"
Sep 07 21:20:31.943713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943554    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcbbf1b6-5158-4780-a968-758132cdeb21" containerName="main"
Sep 07 21:20:31.943713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943566    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9998df7-d3ec-4cad-8d61-e9be1abde86b" containerName="main"
Sep 07 21:20:31.943713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943575    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e037af6-7064-4324-b688-58973c492608" containerName="revoke-keys"
Sep 07 21:20:31.943713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.943585    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7b273ec-9f6e-47e0-8f40-5f6fbe703996" containerName="payload-processing"
Sep 07 21:20:31.946617 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.946597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:31.949169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.949143    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-15d5e2-dockercfg-xfxpv\""
Sep 07 21:20:31.949302 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.949156    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-15d5e2\""
Sep 07 21:20:31.958368 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:31.957578    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"]
Sep 07 21:20:32.003563 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.003538    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:20:32.014937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.014856    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5\" (UID: \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:32.014937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.014900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r224h\" (UniqueName: \"kubernetes.io/projected/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-kube-api-access-r224h\") pod \"payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5\" (UID: \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:32.116100 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116057    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-credential-socket\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116100 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-data\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116131    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2424e7f-f62f-4680-aaee-8aa767988e45-istiod-ca-cert\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116187    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s8cj\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-kube-api-access-2s8cj\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116219    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-certs\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116311    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-token\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116349    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-podinfo\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116384    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-envoy\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116427    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:32.116624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116450    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-socket\") pod \"b2424e7f-f62f-4680-aaee-8aa767988e45\" (UID: \"b2424e7f-f62f-4680-aaee-8aa767988e45\") "
Sep 07 21:20:32.116624 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116526    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:32.116827 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5\" (UID: \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:32.116827 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116690    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2424e7f-f62f-4680-aaee-8aa767988e45-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:20:32.116827 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116722    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r224h\" (UniqueName: \"kubernetes.io/projected/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-kube-api-access-r224h\") pod \"payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5\" (UID: \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:32.116827 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116747    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-data" (OuterVolumeSpecName: "istio-data") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:32.116827 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116759    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:32.117157 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116876    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-socket\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.117157 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116893    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-credential-socket\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.117157 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116907    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-data\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.117157 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116918    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2424e7f-f62f-4680-aaee-8aa767988e45-istiod-ca-cert\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.117157 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.116929    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-workload-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.117549 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.117527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5\" (UID: \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:32.118698 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.118669    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 21:20:32.118815 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.118709    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-kube-api-access-2s8cj" (OuterVolumeSpecName: "kube-api-access-2s8cj") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "kube-api-access-2s8cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:20:32.118918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.118899    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-token" (OuterVolumeSpecName: "istio-token") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:20:32.118996 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.118931    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b2424e7f-f62f-4680-aaee-8aa767988e45" (UID: "b2424e7f-f62f-4680-aaee-8aa767988e45"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:20:32.124439 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.124410    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r224h\" (UniqueName: \"kubernetes.io/projected/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-kube-api-access-r224h\") pod \"payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5\" (UID: \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:32.217821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.217791    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s8cj\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-kube-api-access-2s8cj\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.217821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.217818    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-token\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.217821 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.217828    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-podinfo\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.218064 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.217836    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2424e7f-f62f-4680-aaee-8aa767988e45-istio-envoy\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:20:32.263885 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.263861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:32.469901 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.469867    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerID="7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2" exitCode=0
Sep 07 21:20:32.470107 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.469913    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"
Sep 07 21:20:32.470107 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.469941    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" event={"ID":"b2424e7f-f62f-4680-aaee-8aa767988e45","Type":"ContainerDied","Data":"7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2"}
Sep 07 21:20:32.470107 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.470017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4" event={"ID":"b2424e7f-f62f-4680-aaee-8aa767988e45","Type":"ContainerDied","Data":"cdf342782be0c2a031da77c2eace1cc871e8026eafa56aff5324e63a0a1ad0a8"}
Sep 07 21:20:32.470107 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.470050    2565 scope.go:117] "RemoveContainer" containerID="7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2"
Sep 07 21:20:32.478797 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.478778    2565 scope.go:117] "RemoveContainer" containerID="35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e"
Sep 07 21:20:32.487126 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.487103    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"]
Sep 07 21:20:32.487402 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.487384    2565 scope.go:117] "RemoveContainer" containerID="7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2"
Sep 07 21:20:32.487665 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:20:32.487646    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2\": container with ID starting with 7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2 not found: ID does not exist" containerID="7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2"
Sep 07 21:20:32.487712 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.487673    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2"} err="failed to get container status \"7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2\": rpc error: code = NotFound desc = could not find container \"7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2\": container with ID starting with 7b5c5aa89c793f4cf3e7eec771659933306542cb9f0bdb2fcdef457d11db08a2 not found: ID does not exist"
Sep 07 21:20:32.487712 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.487688    2565 scope.go:117] "RemoveContainer" containerID="35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e"
Sep 07 21:20:32.487935 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:20:32.487919    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e\": container with ID starting with 35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e not found: ID does not exist" containerID="35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e"
Sep 07 21:20:32.487995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.487938    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e"} err="failed to get container status \"35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e\": rpc error: code = NotFound desc = could not find container \"35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e\": container with ID starting with 35d774dab4a2dd02630c9291cf23aad2bc461481b97336ab5e0037cb925d192e not found: ID does not exist"
Sep 07 21:20:32.490906 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.490885    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3102b2-openshift-default-9db689dd-c95p4"]
Sep 07 21:20:32.533193 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:32.533173    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"]
Sep 07 21:20:32.535264 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:20:32.535240    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f9e1b0_fb24_478d_b574_5f0669fa4a4f.slice/crio-f9ae212237284b0ef083537c79c3dae08444e6ca4a2b6175e40fdc35724e1584 WatchSource:0}: Error finding container f9ae212237284b0ef083537c79c3dae08444e6ca4a2b6175e40fdc35724e1584: Status 404 returned error can't find the container with id f9ae212237284b0ef083537c79c3dae08444e6ca4a2b6175e40fdc35724e1584
Sep 07 21:20:33.477739 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:33.477698    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5" event={"ID":"68f9e1b0-fb24-478d-b574-5f0669fa4a4f","Type":"ContainerStarted","Data":"5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035"}
Sep 07 21:20:33.478198 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:33.477759    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5" event={"ID":"68f9e1b0-fb24-478d-b574-5f0669fa4a4f","Type":"ContainerStarted","Data":"f9ae212237284b0ef083537c79c3dae08444e6ca4a2b6175e40fdc35724e1584"}
Sep 07 21:20:33.478198 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:33.477798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:33.494153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:33.494099    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5" podStartSLOduration=2.494083032 podStartE2EDuration="2.494083032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:20:33.492593669 +0000 UTC m=+1125.574121011" watchObservedRunningTime="2026-09-07 21:20:33.494083032 +0000 UTC m=+1125.575610373"
Sep 07 21:20:34.364566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:34.364530    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" path="/var/lib/kubelet/pods/b2424e7f-f62f-4680-aaee-8aa767988e45/volumes"
Sep 07 21:20:40.561953 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:40.561919    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:20:42.286612 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.286571    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"]
Sep 07 21:20:42.287097 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.287064    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy"
Sep 07 21:20:42.287097 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.287083    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy"
Sep 07 21:20:42.287219 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.287198    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy"
Sep 07 21:20:42.287219 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.287214    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy"
Sep 07 21:20:42.291351 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.291329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:42.293457 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.293434    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d03ead-dockercfg-xllcx\""
Sep 07 21:20:42.293576 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.293460    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d03ead\""
Sep 07 21:20:42.303453 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.303085    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"]
Sep 07 21:20:42.404820 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.404778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a44415-010b-4b3e-9057-a74bc15af9a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb\" (UID: \"f7a44415-010b-4b3e-9057-a74bc15af9a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:42.405034 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.404846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbw76\" (UniqueName: \"kubernetes.io/projected/f7a44415-010b-4b3e-9057-a74bc15af9a7-kube-api-access-hbw76\") pod \"payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb\" (UID: \"f7a44415-010b-4b3e-9057-a74bc15af9a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:42.505604 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.505562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a44415-010b-4b3e-9057-a74bc15af9a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb\" (UID: \"f7a44415-010b-4b3e-9057-a74bc15af9a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:42.505774 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.505701    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbw76\" (UniqueName: \"kubernetes.io/projected/f7a44415-010b-4b3e-9057-a74bc15af9a7-kube-api-access-hbw76\") pod \"payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb\" (UID: \"f7a44415-010b-4b3e-9057-a74bc15af9a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:42.506329 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.506309    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a44415-010b-4b3e-9057-a74bc15af9a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb\" (UID: \"f7a44415-010b-4b3e-9057-a74bc15af9a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:42.513467 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.513441    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbw76\" (UniqueName: \"kubernetes.io/projected/f7a44415-010b-4b3e-9057-a74bc15af9a7-kube-api-access-hbw76\") pod \"payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb\" (UID: \"f7a44415-010b-4b3e-9057-a74bc15af9a7\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:42.605208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.605180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:42.872721 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:42.872697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"]
Sep 07 21:20:42.874953 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:20:42.874922    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a44415_010b_4b3e_9057_a74bc15af9a7.slice/crio-545eff32e78c4e09761e1a9fe05b97baea7f1808b8b709584c7173e235835b0c WatchSource:0}: Error finding container 545eff32e78c4e09761e1a9fe05b97baea7f1808b8b709584c7173e235835b0c: Status 404 returned error can't find the container with id 545eff32e78c4e09761e1a9fe05b97baea7f1808b8b709584c7173e235835b0c
Sep 07 21:20:43.525597 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:43.525562    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb" event={"ID":"f7a44415-010b-4b3e-9057-a74bc15af9a7","Type":"ContainerStarted","Data":"0ba9dac9deed910a13402106669c8d3068b7a6c184ec02977dd868b55159ed49"}
Sep 07 21:20:43.526118 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:43.525622    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb" event={"ID":"f7a44415-010b-4b3e-9057-a74bc15af9a7","Type":"ContainerStarted","Data":"545eff32e78c4e09761e1a9fe05b97baea7f1808b8b709584c7173e235835b0c"}
Sep 07 21:20:43.526118 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:43.525651    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:20:43.541712 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:43.541646    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb" podStartSLOduration=1.541631276 podStartE2EDuration="1.541631276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:20:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:20:43.5400832 +0000 UTC m=+1135.621610543" watchObservedRunningTime="2026-09-07 21:20:43.541631276 +0000 UTC m=+1135.623158617"
Sep 07 21:20:44.486136 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:44.486106    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:20:51.462646 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:51.462613    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:20:54.534505 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:20:54.534468    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:21:00.451007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.450957    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"]
Sep 07 21:21:00.451377 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.451301    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy"
Sep 07 21:21:00.451377 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.451312    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2424e7f-f62f-4680-aaee-8aa767988e45" containerName="istio-proxy"
Sep 07 21:21:00.454337 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.454320    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:00.456728 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.456706    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-7fa21f\""
Sep 07 21:21:00.465570 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.465547    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"]
Sep 07 21:21:00.565124 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.565090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmbcr\" (UniqueName: \"kubernetes.io/projected/0a318069-92cc-46b4-add9-3292559a9172-kube-api-access-dmbcr\") pod \"payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p\" (UID: \"0a318069-92cc-46b4-add9-3292559a9172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:00.565284 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.565140    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a318069-92cc-46b4-add9-3292559a9172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p\" (UID: \"0a318069-92cc-46b4-add9-3292559a9172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:00.665759 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.665724    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a318069-92cc-46b4-add9-3292559a9172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p\" (UID: \"0a318069-92cc-46b4-add9-3292559a9172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:00.665930 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.665802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmbcr\" (UniqueName: \"kubernetes.io/projected/0a318069-92cc-46b4-add9-3292559a9172-kube-api-access-dmbcr\") pod \"payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p\" (UID: \"0a318069-92cc-46b4-add9-3292559a9172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:00.666521 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.666501    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a318069-92cc-46b4-add9-3292559a9172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p\" (UID: \"0a318069-92cc-46b4-add9-3292559a9172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:00.673594 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.673566    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmbcr\" (UniqueName: \"kubernetes.io/projected/0a318069-92cc-46b4-add9-3292559a9172-kube-api-access-dmbcr\") pod \"payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p\" (UID: \"0a318069-92cc-46b4-add9-3292559a9172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:00.765052 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:00.764987    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:01.033141 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:01.033117    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"]
Sep 07 21:21:01.035187 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:01.035157    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a318069_92cc_46b4_add9_3292559a9172.slice/crio-173f2e6a81826bad86c0ec94eb59c2eb47c2d21b0cd1579cfc7a262eb28d508e WatchSource:0}: Error finding container 173f2e6a81826bad86c0ec94eb59c2eb47c2d21b0cd1579cfc7a262eb28d508e: Status 404 returned error can't find the container with id 173f2e6a81826bad86c0ec94eb59c2eb47c2d21b0cd1579cfc7a262eb28d508e
Sep 07 21:21:01.612440 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:01.612404    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p" event={"ID":"0a318069-92cc-46b4-add9-3292559a9172","Type":"ContainerStarted","Data":"9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186"}
Sep 07 21:21:01.612806 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:01.612477    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p" event={"ID":"0a318069-92cc-46b4-add9-3292559a9172","Type":"ContainerStarted","Data":"173f2e6a81826bad86c0ec94eb59c2eb47c2d21b0cd1579cfc7a262eb28d508e"}
Sep 07 21:21:01.612806 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:01.612499    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:01.628077 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:01.628031    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p" podStartSLOduration=1.6280166619999998 podStartE2EDuration="1.628016662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:21:01.62669507 +0000 UTC m=+1153.708222413" watchObservedRunningTime="2026-09-07 21:21:01.628016662 +0000 UTC m=+1153.709544003"
Sep 07 21:21:02.965187 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:02.965152    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:21:10.376904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.376868    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"]
Sep 07 21:21:10.381321 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.381301    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:10.383789 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.383768    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b9edc3\""
Sep 07 21:21:10.392043 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.392015    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"]
Sep 07 21:21:10.427351 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.427317    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"]
Sep 07 21:21:10.430655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.430639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:10.433147 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.433127    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b9edc3-dockercfg-xvbnf\""
Sep 07 21:21:10.442175 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.442150    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"]
Sep 07 21:21:10.551828 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.551787    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgtkk\" (UniqueName: \"kubernetes.io/projected/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-kube-api-access-vgtkk\") pod \"payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2\" (UID: \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:10.552027 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.551838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2\" (UID: \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:10.552027 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.551921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1c3243-4dd2-41bb-b875-0654603152ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9\" (UID: \"4a1c3243-4dd2-41bb-b875-0654603152ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:10.552027 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.551954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr658\" (UniqueName: \"kubernetes.io/projected/4a1c3243-4dd2-41bb-b875-0654603152ac-kube-api-access-dr658\") pod \"payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9\" (UID: \"4a1c3243-4dd2-41bb-b875-0654603152ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:10.653328 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.653227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgtkk\" (UniqueName: \"kubernetes.io/projected/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-kube-api-access-vgtkk\") pod \"payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2\" (UID: \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:10.653328 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.653309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2\" (UID: \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:10.653576 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.653364    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1c3243-4dd2-41bb-b875-0654603152ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9\" (UID: \"4a1c3243-4dd2-41bb-b875-0654603152ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:10.653576 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.653394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr658\" (UniqueName: \"kubernetes.io/projected/4a1c3243-4dd2-41bb-b875-0654603152ac-kube-api-access-dr658\") pod \"payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9\" (UID: \"4a1c3243-4dd2-41bb-b875-0654603152ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:10.654150 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.654125    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2\" (UID: \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:10.654247 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.654170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1c3243-4dd2-41bb-b875-0654603152ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9\" (UID: \"4a1c3243-4dd2-41bb-b875-0654603152ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:10.661156 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.661129    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgtkk\" (UniqueName: \"kubernetes.io/projected/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-kube-api-access-vgtkk\") pod \"payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2\" (UID: \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:10.661254 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.661170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr658\" (UniqueName: \"kubernetes.io/projected/4a1c3243-4dd2-41bb-b875-0654603152ac-kube-api-access-dr658\") pod \"payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9\" (UID: \"4a1c3243-4dd2-41bb-b875-0654603152ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:10.692816 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.692737    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:10.741334 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.741096    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:10.980515 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:10.980489    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"]
Sep 07 21:21:10.982609 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:10.982587    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1c3243_4dd2_41bb_b875_0654603152ac.slice/crio-75010fe9879f3c8fedb12636e6d2524b7a8a850e9d37498d59c99f84075d6f88 WatchSource:0}: Error finding container 75010fe9879f3c8fedb12636e6d2524b7a8a850e9d37498d59c99f84075d6f88: Status 404 returned error can't find the container with id 75010fe9879f3c8fedb12636e6d2524b7a8a850e9d37498d59c99f84075d6f88
Sep 07 21:21:11.034751 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.034717    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"]
Sep 07 21:21:11.038097 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:11.038073    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d42e4ee_9433_4bf7_9a70_c9281d6a81c2.slice/crio-73de537fa57a7d2a7dea6e4336633ad34862cac808e0c1f23549e17c3a0dc801 WatchSource:0}: Error finding container 73de537fa57a7d2a7dea6e4336633ad34862cac808e0c1f23549e17c3a0dc801: Status 404 returned error can't find the container with id 73de537fa57a7d2a7dea6e4336633ad34862cac808e0c1f23549e17c3a0dc801
Sep 07 21:21:11.663888 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.663848    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2" event={"ID":"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2","Type":"ContainerStarted","Data":"3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3"}
Sep 07 21:21:11.664461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.663944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2" event={"ID":"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2","Type":"ContainerStarted","Data":"73de537fa57a7d2a7dea6e4336633ad34862cac808e0c1f23549e17c3a0dc801"}
Sep 07 21:21:11.664461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.663995    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:11.665692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.665655    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9" event={"ID":"4a1c3243-4dd2-41bb-b875-0654603152ac","Type":"ContainerStarted","Data":"ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c"}
Sep 07 21:21:11.665828 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.665700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9" event={"ID":"4a1c3243-4dd2-41bb-b875-0654603152ac","Type":"ContainerStarted","Data":"75010fe9879f3c8fedb12636e6d2524b7a8a850e9d37498d59c99f84075d6f88"}
Sep 07 21:21:11.665828 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.665720    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:11.679226 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.679177    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2" podStartSLOduration=1.679160072 podStartE2EDuration="1.679160072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:21:11.678437526 +0000 UTC m=+1163.759964881" watchObservedRunningTime="2026-09-07 21:21:11.679160072 +0000 UTC m=+1163.760687415"
Sep 07 21:21:11.694417 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:11.694369    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9" podStartSLOduration=1.6943552579999999 podStartE2EDuration="1.694355258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:21:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:21:11.692667949 +0000 UTC m=+1163.774195290" watchObservedRunningTime="2026-09-07 21:21:11.694355258 +0000 UTC m=+1163.775882598"
Sep 07 21:21:12.619518 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:12.619487    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:21:22.672497 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:22.672466    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:21:22.672891 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:22.672636    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:21:23.162787 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:23.162749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:21:34.248676 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.248640    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:34.252286 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.252269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:34.255199 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.255171    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6qlgr\""
Sep 07 21:21:34.255774 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.255749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:34.352895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.352864    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf2qb\" (UniqueName: \"kubernetes.io/projected/42f66ee0-d337-453a-8c12-f2b916248d3e-kube-api-access-gf2qb\") pod \"test-curl-7493-139896624661680\" (UID: \"42f66ee0-d337-453a-8c12-f2b916248d3e\") " pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:34.454291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.454258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf2qb\" (UniqueName: \"kubernetes.io/projected/42f66ee0-d337-453a-8c12-f2b916248d3e-kube-api-access-gf2qb\") pod \"test-curl-7493-139896624661680\" (UID: \"42f66ee0-d337-453a-8c12-f2b916248d3e\") " pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:34.461788 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.461767    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf2qb\" (UniqueName: \"kubernetes.io/projected/42f66ee0-d337-453a-8c12-f2b916248d3e-kube-api-access-gf2qb\") pod \"test-curl-7493-139896624661680\" (UID: \"42f66ee0-d337-453a-8c12-f2b916248d3e\") " pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:34.563343 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.563270    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:34.830383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.830359    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:34.832122 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:34.832095    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42f66ee0_d337_453a_8c12_f2b916248d3e.slice/crio-8662a8657c91042cf5d2b28c3cd15f1f7069021d5d420ca78eeb4323305aa776 WatchSource:0}: Error finding container 8662a8657c91042cf5d2b28c3cd15f1f7069021d5d420ca78eeb4323305aa776: Status 404 returned error can't find the container with id 8662a8657c91042cf5d2b28c3cd15f1f7069021d5d420ca78eeb4323305aa776
Sep 07 21:21:34.833679 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:34.833658    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 21:21:35.789950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:35.789859    2565 generic.go:356] "Generic (PLEG): container finished" podID="42f66ee0-d337-453a-8c12-f2b916248d3e" containerID="ed558d6cb6eeace3e6d91c87f77f051215ca48f88970734b706bd0720d5f78f0" exitCode=0
Sep 07 21:21:35.789950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:35.789919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7493-139896624661680" event={"ID":"42f66ee0-d337-453a-8c12-f2b916248d3e","Type":"ContainerDied","Data":"ed558d6cb6eeace3e6d91c87f77f051215ca48f88970734b706bd0720d5f78f0"}
Sep 07 21:21:35.790370 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:35.789952    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7493-139896624661680" event={"ID":"42f66ee0-d337-453a-8c12-f2b916248d3e","Type":"ContainerStarted","Data":"8662a8657c91042cf5d2b28c3cd15f1f7069021d5d420ca78eeb4323305aa776"}
Sep 07 21:21:36.913567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:36.913534    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:36.945884 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:36.945856    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7493-139896624661680_42f66ee0-d337-453a-8c12-f2b916248d3e/test-curl-7493-139896624661680/0.log"
Sep 07 21:21:36.966780 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:36.966750    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:36.972462 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:36.972440    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:36.984043 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:36.984024    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf2qb\" (UniqueName: \"kubernetes.io/projected/42f66ee0-d337-453a-8c12-f2b916248d3e-kube-api-access-gf2qb\") pod \"42f66ee0-d337-453a-8c12-f2b916248d3e\" (UID: \"42f66ee0-d337-453a-8c12-f2b916248d3e\") "
Sep 07 21:21:36.986057 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:36.986029    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42f66ee0-d337-453a-8c12-f2b916248d3e-kube-api-access-gf2qb" (OuterVolumeSpecName: "kube-api-access-gf2qb") pod "42f66ee0-d337-453a-8c12-f2b916248d3e" (UID: "42f66ee0-d337-453a-8c12-f2b916248d3e"). InnerVolumeSpecName "kube-api-access-gf2qb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:21:37.085399 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.085366    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf2qb\" (UniqueName: \"kubernetes.io/projected/42f66ee0-d337-453a-8c12-f2b916248d3e-kube-api-access-gf2qb\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:21:37.086121 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.086100    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:37.086466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.086455    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f66ee0-d337-453a-8c12-f2b916248d3e" containerName="test-curl-7493-139896624661680"
Sep 07 21:21:37.086508 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.086469    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f66ee0-d337-453a-8c12-f2b916248d3e" containerName="test-curl-7493-139896624661680"
Sep 07 21:21:37.086555 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.086547    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f66ee0-d337-453a-8c12-f2b916248d3e" containerName="test-curl-7493-139896624661680"
Sep 07 21:21:37.089550 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.089534    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:37.095189 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.095162    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:37.186408 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.186375    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mlwz\" (UniqueName: \"kubernetes.io/projected/8e304e85-a925-4e60-9474-0a308f9560a5-kube-api-access-7mlwz\") pod \"test-curl-7493-139896624661680\" (UID: \"8e304e85-a925-4e60-9474-0a308f9560a5\") " pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:37.287286 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.287241    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mlwz\" (UniqueName: \"kubernetes.io/projected/8e304e85-a925-4e60-9474-0a308f9560a5-kube-api-access-7mlwz\") pod \"test-curl-7493-139896624661680\" (UID: \"8e304e85-a925-4e60-9474-0a308f9560a5\") " pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:37.295691 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.295665    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mlwz\" (UniqueName: \"kubernetes.io/projected/8e304e85-a925-4e60-9474-0a308f9560a5-kube-api-access-7mlwz\") pod \"test-curl-7493-139896624661680\" (UID: \"8e304e85-a925-4e60-9474-0a308f9560a5\") " pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:37.400104 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.400012    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:37.668401 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.668378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:37.670743 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:37.670714    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e304e85_a925_4e60_9474_0a308f9560a5.slice/crio-b78614feadd35b86f3836a9a1c82472e04e71198fdf1808ba8f9a23c887363a1 WatchSource:0}: Error finding container b78614feadd35b86f3836a9a1c82472e04e71198fdf1808ba8f9a23c887363a1: Status 404 returned error can't find the container with id b78614feadd35b86f3836a9a1c82472e04e71198fdf1808ba8f9a23c887363a1
Sep 07 21:21:37.801489 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.801457    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:37.801661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.801494    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8662a8657c91042cf5d2b28c3cd15f1f7069021d5d420ca78eeb4323305aa776"
Sep 07 21:21:37.802840 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.802809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7493-139896624661680" event={"ID":"8e304e85-a925-4e60-9474-0a308f9560a5","Type":"ContainerStarted","Data":"b78614feadd35b86f3836a9a1c82472e04e71198fdf1808ba8f9a23c887363a1"}
Sep 07 21:21:37.812925 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:37.812900    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7493-139896624661680" oldPodUID="42f66ee0-d337-453a-8c12-f2b916248d3e" podUID="8e304e85-a925-4e60-9474-0a308f9560a5"
Sep 07 21:21:38.364995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:38.364947    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f66ee0-d337-453a-8c12-f2b916248d3e" path="/var/lib/kubelet/pods/42f66ee0-d337-453a-8c12-f2b916248d3e/volumes"
Sep 07 21:21:38.809509 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:38.809410    2565 generic.go:356] "Generic (PLEG): container finished" podID="8e304e85-a925-4e60-9474-0a308f9560a5" containerID="dd9d6a0ab00844d7fd4a7e7af68a03363ccd70451793ec0855c0e1379f1d6c62" exitCode=0
Sep 07 21:21:38.809655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:38.809498    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7493-139896624661680" event={"ID":"8e304e85-a925-4e60-9474-0a308f9560a5","Type":"ContainerDied","Data":"dd9d6a0ab00844d7fd4a7e7af68a03363ccd70451793ec0855c0e1379f1d6c62"}
Sep 07 21:21:39.941735 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:39.941706    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:39.963173 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:39.963150    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7493-139896624661680_8e304e85-a925-4e60-9474-0a308f9560a5/test-curl-7493-139896624661680/0.log"
Sep 07 21:21:39.987569 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:39.987545    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:39.988924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:39.988904    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7493-139896624661680"]
Sep 07 21:21:40.007937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.007913    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mlwz\" (UniqueName: \"kubernetes.io/projected/8e304e85-a925-4e60-9474-0a308f9560a5-kube-api-access-7mlwz\") pod \"8e304e85-a925-4e60-9474-0a308f9560a5\" (UID: \"8e304e85-a925-4e60-9474-0a308f9560a5\") "
Sep 07 21:21:40.009885 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.009860    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e304e85-a925-4e60-9474-0a308f9560a5-kube-api-access-7mlwz" (OuterVolumeSpecName: "kube-api-access-7mlwz") pod "8e304e85-a925-4e60-9474-0a308f9560a5" (UID: "8e304e85-a925-4e60-9474-0a308f9560a5"). InnerVolumeSpecName "kube-api-access-7mlwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:21:40.105664 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.105632    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:40.106073 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.106056    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e304e85-a925-4e60-9474-0a308f9560a5" containerName="test-curl-7493-139896624661680"
Sep 07 21:21:40.106163 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.106076    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e304e85-a925-4e60-9474-0a308f9560a5" containerName="test-curl-7493-139896624661680"
Sep 07 21:21:40.106213 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.106188    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e304e85-a925-4e60-9474-0a308f9560a5" containerName="test-curl-7493-139896624661680"
Sep 07 21:21:40.109205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.109185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:40.109421 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.109399    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mlwz\" (UniqueName: \"kubernetes.io/projected/8e304e85-a925-4e60-9474-0a308f9560a5-kube-api-access-7mlwz\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:21:40.113079 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.113058    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:40.210031 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.210001    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg55t\" (UniqueName: \"kubernetes.io/projected/f8eff87a-c1b6-4233-b6bf-3700f831feb0-kube-api-access-fg55t\") pod \"test-curl-7493-139896624662256\" (UID: \"f8eff87a-c1b6-4233-b6bf-3700f831feb0\") " pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:40.310611 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.310574    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg55t\" (UniqueName: \"kubernetes.io/projected/f8eff87a-c1b6-4233-b6bf-3700f831feb0-kube-api-access-fg55t\") pod \"test-curl-7493-139896624662256\" (UID: \"f8eff87a-c1b6-4233-b6bf-3700f831feb0\") " pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:40.318851 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.318825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg55t\" (UniqueName: \"kubernetes.io/projected/f8eff87a-c1b6-4233-b6bf-3700f831feb0-kube-api-access-fg55t\") pod \"test-curl-7493-139896624662256\" (UID: \"f8eff87a-c1b6-4233-b6bf-3700f831feb0\") " pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:40.363428 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.363362    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e304e85-a925-4e60-9474-0a308f9560a5" path="/var/lib/kubelet/pods/8e304e85-a925-4e60-9474-0a308f9560a5/volumes"
Sep 07 21:21:40.419459 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.419424    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:40.689091 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.689067    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:40.691156 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:40.691125    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8eff87a_c1b6_4233_b6bf_3700f831feb0.slice/crio-82d6367d0fc65276dd2d053836bb03b451024f2b1a480273ba8cd88bdc5cc175 WatchSource:0}: Error finding container 82d6367d0fc65276dd2d053836bb03b451024f2b1a480273ba8cd88bdc5cc175: Status 404 returned error can't find the container with id 82d6367d0fc65276dd2d053836bb03b451024f2b1a480273ba8cd88bdc5cc175
Sep 07 21:21:40.821267 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.821234    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624661680"
Sep 07 21:21:40.821267 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.821236    2565 scope.go:117] "RemoveContainer" containerID="dd9d6a0ab00844d7fd4a7e7af68a03363ccd70451793ec0855c0e1379f1d6c62"
Sep 07 21:21:40.822635 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:40.822577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7493-139896624662256" event={"ID":"f8eff87a-c1b6-4233-b6bf-3700f831feb0","Type":"ContainerStarted","Data":"82d6367d0fc65276dd2d053836bb03b451024f2b1a480273ba8cd88bdc5cc175"}
Sep 07 21:21:41.830586 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:41.830550    2565 generic.go:356] "Generic (PLEG): container finished" podID="f8eff87a-c1b6-4233-b6bf-3700f831feb0" containerID="0753f11629546b19d365efe3a3162389561d6297ededaec36ce09ddb46132390" exitCode=0
Sep 07 21:21:41.830991 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:41.830635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7493-139896624662256" event={"ID":"f8eff87a-c1b6-4233-b6bf-3700f831feb0","Type":"ContainerDied","Data":"0753f11629546b19d365efe3a3162389561d6297ededaec36ce09ddb46132390"}
Sep 07 21:21:42.352307 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.352271    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"]
Sep 07 21:21:42.355571 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.355550    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.358399 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.358375    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"openshift-service-ca.crt\""
Sep 07 21:21:42.359381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.359360    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"kube-root-ca.crt\""
Sep 07 21:21:42.359512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.359404    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"auth-test-model-15d5e2-kserve-self-signed-certs\""
Sep 07 21:21:42.359512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.359411    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"default-dockercfg-wbbfn\""
Sep 07 21:21:42.366813 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.366790    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"]
Sep 07 21:21:42.430218 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.430192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-dshm\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.430435 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.430398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-home\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.430510 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.430440    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmvn\" (UniqueName: \"kubernetes.io/projected/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-kube-api-access-sqmvn\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.430510 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.430501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-model-cache\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.430641 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.430626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tls-certs\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.430735 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.430721    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tmp-dir\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531368 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531335    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-dshm\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-home\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531415    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmvn\" (UniqueName: \"kubernetes.io/projected/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-kube-api-access-sqmvn\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-model-cache\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tls-certs\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531687 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531525    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tmp-dir\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531822 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531798    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-home\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531842    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tmp-dir\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.531895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.531873    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-model-cache\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.533544 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.533527    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-dshm\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.533853 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.533837    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tls-certs\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.539067 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.539043    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmvn\" (UniqueName: \"kubernetes.io/projected/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-kube-api-access-sqmvn\") pod \"auth-test-model-15d5e2-kserve-6d8d9699-gv75l\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.666885 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.666817    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:42.939428 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.939387    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"]
Sep 07 21:21:42.941919 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:42.941889    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0831cc86_e01d_42f2_8a70_4c8c76fe9f3b.slice/crio-9bc1fd188e56e9224ddf13a4da5751b703fc50cf73b9ddbe219da48314ed13b9 WatchSource:0}: Error finding container 9bc1fd188e56e9224ddf13a4da5751b703fc50cf73b9ddbe219da48314ed13b9: Status 404 returned error can't find the container with id 9bc1fd188e56e9224ddf13a4da5751b703fc50cf73b9ddbe219da48314ed13b9
Sep 07 21:21:42.956129 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.956109    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:42.975767 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.975746    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7493-139896624662256_f8eff87a-c1b6-4233-b6bf-3700f831feb0/test-curl-7493-139896624662256/0.log"
Sep 07 21:21:42.997839 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:42.997819    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:43.002968 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.002937    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:43.036147 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.036123    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg55t\" (UniqueName: \"kubernetes.io/projected/f8eff87a-c1b6-4233-b6bf-3700f831feb0-kube-api-access-fg55t\") pod \"f8eff87a-c1b6-4233-b6bf-3700f831feb0\" (UID: \"f8eff87a-c1b6-4233-b6bf-3700f831feb0\") "
Sep 07 21:21:43.038108 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.038084    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eff87a-c1b6-4233-b6bf-3700f831feb0-kube-api-access-fg55t" (OuterVolumeSpecName: "kube-api-access-fg55t") pod "f8eff87a-c1b6-4233-b6bf-3700f831feb0" (UID: "f8eff87a-c1b6-4233-b6bf-3700f831feb0"). InnerVolumeSpecName "kube-api-access-fg55t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:21:43.116575 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.116541    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:43.117018 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.116999    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8eff87a-c1b6-4233-b6bf-3700f831feb0" containerName="test-curl-7493-139896624662256"
Sep 07 21:21:43.117018 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.117019    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8eff87a-c1b6-4233-b6bf-3700f831feb0" containerName="test-curl-7493-139896624662256"
Sep 07 21:21:43.117168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.117109    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8eff87a-c1b6-4233-b6bf-3700f831feb0" containerName="test-curl-7493-139896624662256"
Sep 07 21:21:43.120072 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.120053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:43.123756 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.123733    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:43.149462 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.149436    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg55t\" (UniqueName: \"kubernetes.io/projected/f8eff87a-c1b6-4233-b6bf-3700f831feb0-kube-api-access-fg55t\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:21:43.250591 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.250518    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nv6q\" (UniqueName: \"kubernetes.io/projected/b56ffa2b-49c2-430e-8e1b-406c262b42e9-kube-api-access-7nv6q\") pod \"test-curl-7493-139896624662256\" (UID: \"b56ffa2b-49c2-430e-8e1b-406c262b42e9\") " pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:43.351856 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.351825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nv6q\" (UniqueName: \"kubernetes.io/projected/b56ffa2b-49c2-430e-8e1b-406c262b42e9-kube-api-access-7nv6q\") pod \"test-curl-7493-139896624662256\" (UID: \"b56ffa2b-49c2-430e-8e1b-406c262b42e9\") " pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:43.359240 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.359216    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nv6q\" (UniqueName: \"kubernetes.io/projected/b56ffa2b-49c2-430e-8e1b-406c262b42e9-kube-api-access-7nv6q\") pod \"test-curl-7493-139896624662256\" (UID: \"b56ffa2b-49c2-430e-8e1b-406c262b42e9\") " pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:43.451214 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.451190    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:43.717579 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.717361    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:43.720141 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:43.720100    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb56ffa2b_49c2_430e_8e1b_406c262b42e9.slice/crio-c87a5a75eadebea10e444a11decd5b38c710e355a323aa243281c8ae88c5ae65 WatchSource:0}: Error finding container c87a5a75eadebea10e444a11decd5b38c710e355a323aa243281c8ae88c5ae65: Status 404 returned error can't find the container with id c87a5a75eadebea10e444a11decd5b38c710e355a323aa243281c8ae88c5ae65
Sep 07 21:21:43.847803 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.847768    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7493-139896624662256" event={"ID":"b56ffa2b-49c2-430e-8e1b-406c262b42e9","Type":"ContainerStarted","Data":"c87a5a75eadebea10e444a11decd5b38c710e355a323aa243281c8ae88c5ae65"}
Sep 07 21:21:43.849439 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.849411    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l" event={"ID":"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b","Type":"ContainerStarted","Data":"f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4"}
Sep 07 21:21:43.849593 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.849448    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l" event={"ID":"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b","Type":"ContainerStarted","Data":"9bc1fd188e56e9224ddf13a4da5751b703fc50cf73b9ddbe219da48314ed13b9"}
Sep 07 21:21:43.854213 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.854191    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d6367d0fc65276dd2d053836bb03b451024f2b1a480273ba8cd88bdc5cc175"
Sep 07 21:21:43.854301 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.854216    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:43.867217 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.867173    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l" podStartSLOduration=1.8671595029999999 podStartE2EDuration="1.867159503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:21:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:21:43.866312301 +0000 UTC m=+1195.947839643" watchObservedRunningTime="2026-09-07 21:21:43.867159503 +0000 UTC m=+1195.948686845"
Sep 07 21:21:43.869049 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:43.869025    2565 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7493-139896624662256" oldPodUID="f8eff87a-c1b6-4233-b6bf-3700f831feb0" podUID="b56ffa2b-49c2-430e-8e1b-406c262b42e9"
Sep 07 21:21:44.364478 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:44.364450    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8eff87a-c1b6-4233-b6bf-3700f831feb0" path="/var/lib/kubelet/pods/f8eff87a-c1b6-4233-b6bf-3700f831feb0/volumes"
Sep 07 21:21:44.861266 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:44.861229    2565 generic.go:356] "Generic (PLEG): container finished" podID="b56ffa2b-49c2-430e-8e1b-406c262b42e9" containerID="f37061eb7e7b17263780d3196d79ba4575462f8826c77a68a48b965ca78725dd" exitCode=0
Sep 07 21:21:44.861449 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:44.861307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7493-139896624662256" event={"ID":"b56ffa2b-49c2-430e-8e1b-406c262b42e9","Type":"ContainerDied","Data":"f37061eb7e7b17263780d3196d79ba4575462f8826c77a68a48b965ca78725dd"}
Sep 07 21:21:46.003245 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.003223    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:46.024689 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.024662    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7493-139896624662256_b56ffa2b-49c2-430e-8e1b-406c262b42e9/test-curl-7493-139896624662256/0.log"
Sep 07 21:21:46.046566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.046540    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:46.050466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.050444    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7493-139896624662256"]
Sep 07 21:21:46.176241 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.176146    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nv6q\" (UniqueName: \"kubernetes.io/projected/b56ffa2b-49c2-430e-8e1b-406c262b42e9-kube-api-access-7nv6q\") pod \"b56ffa2b-49c2-430e-8e1b-406c262b42e9\" (UID: \"b56ffa2b-49c2-430e-8e1b-406c262b42e9\") "
Sep 07 21:21:46.178233 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.178206    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56ffa2b-49c2-430e-8e1b-406c262b42e9-kube-api-access-7nv6q" (OuterVolumeSpecName: "kube-api-access-7nv6q") pod "b56ffa2b-49c2-430e-8e1b-406c262b42e9" (UID: "b56ffa2b-49c2-430e-8e1b-406c262b42e9"). InnerVolumeSpecName "kube-api-access-7nv6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:21:46.277801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.277750    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nv6q\" (UniqueName: \"kubernetes.io/projected/b56ffa2b-49c2-430e-8e1b-406c262b42e9-kube-api-access-7nv6q\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:21:46.363779 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.363743    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b56ffa2b-49c2-430e-8e1b-406c262b42e9" path="/var/lib/kubelet/pods/b56ffa2b-49c2-430e-8e1b-406c262b42e9/volumes"
Sep 07 21:21:46.874176 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.874147    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7493-139896624662256"
Sep 07 21:21:46.874350 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:46.874147    2565 scope.go:117] "RemoveContainer" containerID="f37061eb7e7b17263780d3196d79ba4575462f8826c77a68a48b965ca78725dd"
Sep 07 21:21:47.560055 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:47.560017    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:21:48.500206 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:48.500167    2565 scope.go:117] "RemoveContainer" containerID="0b5692b850c4fe6661005137d1cde4d2c4fde7a370f727ba1a64b078f1ced220"
Sep 07 21:21:48.508970 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:48.508940    2565 scope.go:117] "RemoveContainer" containerID="169389720b18e386ada850aa471c746c2dbb900313337a95252795c4ff4802af"
Sep 07 21:21:51.682659 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.682623    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"]
Sep 07 21:21:51.683067 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.683005    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b56ffa2b-49c2-430e-8e1b-406c262b42e9" containerName="test-curl-7493-139896624662256"
Sep 07 21:21:51.683067 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.683020    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b56ffa2b-49c2-430e-8e1b-406c262b42e9" containerName="test-curl-7493-139896624662256"
Sep 07 21:21:51.683144 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.683085    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b56ffa2b-49c2-430e-8e1b-406c262b42e9" containerName="test-curl-7493-139896624662256"
Sep 07 21:21:51.687612 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.687592    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:51.690784 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.690765    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-424fbbc4\""
Sep 07 21:21:51.690918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.690869    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-424fbbc4-dockercfg-n25x9\""
Sep 07 21:21:51.701317 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.701296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"]
Sep 07 21:21:51.711809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.711785    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"]
Sep 07 21:21:51.715686 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.715665    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:51.725316 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.725296    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"]
Sep 07 21:21:51.823075 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.823049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x\" (UID: \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:51.823222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.823110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8jls\" (UniqueName: \"kubernetes.io/projected/974b1009-5faf-44cf-8d85-203ee50b85c4-kube-api-access-l8jls\") pod \"payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg\" (UID: \"974b1009-5faf-44cf-8d85-203ee50b85c4\") " pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:51.823222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.823158    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974b1009-5faf-44cf-8d85-203ee50b85c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg\" (UID: \"974b1009-5faf-44cf-8d85-203ee50b85c4\") " pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:51.823222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.823192    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd5fm\" (UniqueName: \"kubernetes.io/projected/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-kube-api-access-xd5fm\") pod \"payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x\" (UID: \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:51.924001 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.923944    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x\" (UID: \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:51.924187 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.924025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8jls\" (UniqueName: \"kubernetes.io/projected/974b1009-5faf-44cf-8d85-203ee50b85c4-kube-api-access-l8jls\") pod \"payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg\" (UID: \"974b1009-5faf-44cf-8d85-203ee50b85c4\") " pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:51.924187 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.924055    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974b1009-5faf-44cf-8d85-203ee50b85c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg\" (UID: \"974b1009-5faf-44cf-8d85-203ee50b85c4\") " pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:51.924187 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.924085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd5fm\" (UniqueName: \"kubernetes.io/projected/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-kube-api-access-xd5fm\") pod \"payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x\" (UID: \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:51.930083 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.925007    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x\" (UID: \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:51.930083 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.925297    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974b1009-5faf-44cf-8d85-203ee50b85c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg\" (UID: \"974b1009-5faf-44cf-8d85-203ee50b85c4\") " pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:51.932991 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.932921    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8jls\" (UniqueName: \"kubernetes.io/projected/974b1009-5faf-44cf-8d85-203ee50b85c4-kube-api-access-l8jls\") pod \"payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg\" (UID: \"974b1009-5faf-44cf-8d85-203ee50b85c4\") " pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:51.933166 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.933144    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd5fm\" (UniqueName: \"kubernetes.io/projected/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-kube-api-access-xd5fm\") pod \"payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x\" (UID: \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:51.997854 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:51.997828    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:52.026604 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.026573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:52.266873 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.266847    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"]
Sep 07 21:21:52.268570 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:52.268545    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd39867b3_63b4_44ba_9fae_a7feb7be9cfe.slice/crio-13ceb9ee3c45c9a3b59f4dd9b4bb119d00a201d7cb2a8a546165a0d2e7c60cf4 WatchSource:0}: Error finding container 13ceb9ee3c45c9a3b59f4dd9b4bb119d00a201d7cb2a8a546165a0d2e7c60cf4: Status 404 returned error can't find the container with id 13ceb9ee3c45c9a3b59f4dd9b4bb119d00a201d7cb2a8a546165a0d2e7c60cf4
Sep 07 21:21:52.303950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.303923    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"]
Sep 07 21:21:52.305311 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:21:52.305288    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod974b1009_5faf_44cf_8d85_203ee50b85c4.slice/crio-a7babda1cd166beabe4db040415e7ec6df59345e71095b3043a7a29baa05a38b WatchSource:0}: Error finding container a7babda1cd166beabe4db040415e7ec6df59345e71095b3043a7a29baa05a38b: Status 404 returned error can't find the container with id a7babda1cd166beabe4db040415e7ec6df59345e71095b3043a7a29baa05a38b
Sep 07 21:21:52.667221 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.667191    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:52.667386 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.667240    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:52.686196 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.686170    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:52.907736 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.907696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg" event={"ID":"974b1009-5faf-44cf-8d85-203ee50b85c4","Type":"ContainerStarted","Data":"72d24029f31f459f6baf80427be2661f61ca6c382c11ade860bf4783ac65e340"}
Sep 07 21:21:52.907949 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.907758    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg" event={"ID":"974b1009-5faf-44cf-8d85-203ee50b85c4","Type":"ContainerStarted","Data":"a7babda1cd166beabe4db040415e7ec6df59345e71095b3043a7a29baa05a38b"}
Sep 07 21:21:52.907949 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.907787    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:21:52.909338 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.909309    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x" event={"ID":"d39867b3-63b4-44ba-9fae-a7feb7be9cfe","Type":"ContainerStarted","Data":"ad4913d6ead72bc78d30ee40e71e635dae929ecf63f6cf8871cae41d5a25e308"}
Sep 07 21:21:52.909460 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.909371    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x" event={"ID":"d39867b3-63b4-44ba-9fae-a7feb7be9cfe","Type":"ContainerStarted","Data":"13ceb9ee3c45c9a3b59f4dd9b4bb119d00a201d7cb2a8a546165a0d2e7c60cf4"}
Sep 07 21:21:52.909504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.909458    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:21:52.920655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.920592    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:21:52.923467 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.923428    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg" podStartSLOduration=1.9234134 podStartE2EDuration="1.9234134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:21:52.921606377 +0000 UTC m=+1205.003133731" watchObservedRunningTime="2026-09-07 21:21:52.9234134 +0000 UTC m=+1205.004940741"
Sep 07 21:21:52.936219 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:21:52.936170    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x" podStartSLOduration=1.936155821 podStartE2EDuration="1.936155821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:21:52.935426573 +0000 UTC m=+1205.016953915" watchObservedRunningTime="2026-09-07 21:21:52.936155821 +0000 UTC m=+1205.017683227"
Sep 07 21:22:03.916527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:03.916496    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:22:03.916952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:03.916564    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:22:07.152855 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:07.152816    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:22:25.747634 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.747596    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"]
Sep 07 21:22:25.751513 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.751491    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.754801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.754757    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"kube-root-ca.crt\""
Sep 07 21:22:25.754937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.754763    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"auth-test-model-7fa21f-kserve-self-signed-certs\""
Sep 07 21:22:25.754937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.754862    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"openshift-service-ca.crt\""
Sep 07 21:22:25.754937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.754926    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"default-dockercfg-jpnn6\""
Sep 07 21:22:25.761649 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.761629    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"]
Sep 07 21:22:25.833861 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.833827    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tmp-dir\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.834024 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.833861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-home\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.834024 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.833889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tls-certs\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.834024 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.834011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-model-cache\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.834178 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.834045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjwv\" (UniqueName: \"kubernetes.io/projected/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-kube-api-access-xmjwv\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.834178 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.834095    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-dshm\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.934585 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.934556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tmp-dir\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.934585 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.934587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-home\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.934807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.934606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tls-certs\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.934807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.934645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-model-cache\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.934807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.934676    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjwv\" (UniqueName: \"kubernetes.io/projected/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-kube-api-access-xmjwv\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.934807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.934709    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-dshm\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.935065 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.934957    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-home\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.935124 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.935061    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tmp-dir\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.935165 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.935145    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-model-cache\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.936989 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.936956    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-dshm\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.937189 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.937171    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tls-certs\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:25.942132 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:25.942113    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjwv\" (UniqueName: \"kubernetes.io/projected/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-kube-api-access-xmjwv\") pod \"auth-test-model-7fa21f-kserve-54dfb64875-gthdr\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:26.063225 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:26.063184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:26.334067 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:26.334038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"]
Sep 07 21:22:26.336077 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:22:26.336050    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode90baf1a_2cd9_4501_b7fa_c13cd6a2f6ed.slice/crio-c3268dd0a22470340cb1d48340094d146b35fa0a9fa3e63c8162a078cf3ca4dd WatchSource:0}: Error finding container c3268dd0a22470340cb1d48340094d146b35fa0a9fa3e63c8162a078cf3ca4dd: Status 404 returned error can't find the container with id c3268dd0a22470340cb1d48340094d146b35fa0a9fa3e63c8162a078cf3ca4dd
Sep 07 21:22:27.080368 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:27.080327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr" event={"ID":"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed","Type":"ContainerStarted","Data":"0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be"}
Sep 07 21:22:27.080368 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:27.080370    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr" event={"ID":"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed","Type":"ContainerStarted","Data":"c3268dd0a22470340cb1d48340094d146b35fa0a9fa3e63c8162a078cf3ca4dd"}
Sep 07 21:22:27.096030 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:27.095948    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr" podStartSLOduration=2.095932948 podStartE2EDuration="2.095932948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:22:27.095215284 +0000 UTC m=+1239.176742625" watchObservedRunningTime="2026-09-07 21:22:27.095932948 +0000 UTC m=+1239.177460289"
Sep 07 21:22:30.769820 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:30.769778    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:22:36.063652 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:36.063616    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:36.064086 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:36.063674    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:36.076391 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:36.076359    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:36.140371 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:36.140343    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:22:48.541787 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:48.541750    2565 scope.go:117] "RemoveContainer" containerID="487aac6e20b7eac46a4b6a89e926a6025b22857decabbc56599f41a67506dfd8"
Sep 07 21:22:48.549999 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:48.549979    2565 scope.go:117] "RemoveContainer" containerID="6faf1653f20125610bbfabbed36f532182364e9cff0aa95ee5c3fc0aa94f3299"
Sep 07 21:22:49.248996 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:22:49.248945    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:23:03.749654 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:03.749614    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:23:09.716308 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:09.716271    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"]
Sep 07 21:23:09.719729 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:09.719712    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"
Sep 07 21:23:09.721975 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:09.721935    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:23:09.721975 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:09.721942    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:23:09.726685 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:09.726661    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"]
Sep 07 21:23:09.825398 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:09.825364    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84947cf6-26ac-41df-b008-4a4e6c9e5af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx\" (UID: \"84947cf6-26ac-41df-b008-4a4e6c9e5af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"
Sep 07 21:23:09.926486 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:09.926450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84947cf6-26ac-41df-b008-4a4e6c9e5af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx\" (UID: \"84947cf6-26ac-41df-b008-4a4e6c9e5af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"
Sep 07 21:23:09.927099 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:09.927077    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84947cf6-26ac-41df-b008-4a4e6c9e5af5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx\" (UID: \"84947cf6-26ac-41df-b008-4a4e6c9e5af5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"
Sep 07 21:23:10.032581 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:10.032510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"
Sep 07 21:23:10.304067 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:10.303907    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"]
Sep 07 21:23:10.306786 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:23:10.306754    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84947cf6_26ac_41df_b008_4a4e6c9e5af5.slice/crio-83e3d79cc3f93c9c12cd8817e9a0ebd0143576b54da211a62ca27ca055d0658b WatchSource:0}: Error finding container 83e3d79cc3f93c9c12cd8817e9a0ebd0143576b54da211a62ca27ca055d0658b: Status 404 returned error can't find the container with id 83e3d79cc3f93c9c12cd8817e9a0ebd0143576b54da211a62ca27ca055d0658b
Sep 07 21:23:11.303643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:11.303550    2565 generic.go:356] "Generic (PLEG): container finished" podID="84947cf6-26ac-41df-b008-4a4e6c9e5af5" containerID="154d4985f05ff19cb7be37d3aae27cad021c5e2d3d5c7634a35a02f757c6ea65" exitCode=0
Sep 07 21:23:11.303643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:11.303593    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx" event={"ID":"84947cf6-26ac-41df-b008-4a4e6c9e5af5","Type":"ContainerDied","Data":"154d4985f05ff19cb7be37d3aae27cad021c5e2d3d5c7634a35a02f757c6ea65"}
Sep 07 21:23:11.303643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:11.303644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx" event={"ID":"84947cf6-26ac-41df-b008-4a4e6c9e5af5","Type":"ContainerStarted","Data":"83e3d79cc3f93c9c12cd8817e9a0ebd0143576b54da211a62ca27ca055d0658b"}
Sep 07 21:23:12.441692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:12.441668    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"
Sep 07 21:23:12.550682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:12.550654    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84947cf6-26ac-41df-b008-4a4e6c9e5af5-maas-api-service-ca\") pod \"84947cf6-26ac-41df-b008-4a4e6c9e5af5\" (UID: \"84947cf6-26ac-41df-b008-4a4e6c9e5af5\") "
Sep 07 21:23:12.551021 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:12.550996    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84947cf6-26ac-41df-b008-4a4e6c9e5af5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "84947cf6-26ac-41df-b008-4a4e6c9e5af5" (UID: "84947cf6-26ac-41df-b008-4a4e6c9e5af5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:23:12.651620 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:12.651596    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/84947cf6-26ac-41df-b008-4a4e6c9e5af5-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:23:13.316376 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:13.316344    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"
Sep 07 21:23:13.316551 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:13.316378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx" event={"ID":"84947cf6-26ac-41df-b008-4a4e6c9e5af5","Type":"ContainerDied","Data":"83e3d79cc3f93c9c12cd8817e9a0ebd0143576b54da211a62ca27ca055d0658b"}
Sep 07 21:23:13.316551 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:13.316423    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83e3d79cc3f93c9c12cd8817e9a0ebd0143576b54da211a62ca27ca055d0658b"
Sep 07 21:23:21.662750 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:21.662714    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:23:25.205726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.205688    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"]
Sep 07 21:23:25.206469 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.205929    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg" podUID="974b1009-5faf-44cf-8d85-203ee50b85c4" containerName="payload-processing" containerID="cri-o://72d24029f31f459f6baf80427be2661f61ca6c382c11ade860bf4783ac65e340" gracePeriod=30
Sep 07 21:23:25.212622 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.212595    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"]
Sep 07 21:23:25.213013 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.212949    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x" podUID="d39867b3-63b4-44ba-9fae-a7feb7be9cfe" containerName="payload-pre-processing" containerID="cri-o://ad4913d6ead72bc78d30ee40e71e635dae929ecf63f6cf8871cae41d5a25e308" gracePeriod=30
Sep 07 21:23:25.374761 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.374658    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"]
Sep 07 21:23:25.376892 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.376866    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-424fbbc4-4dqqx"]
Sep 07 21:23:25.387037 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.387001    2565 generic.go:356] "Generic (PLEG): container finished" podID="d39867b3-63b4-44ba-9fae-a7feb7be9cfe" containerID="ad4913d6ead72bc78d30ee40e71e635dae929ecf63f6cf8871cae41d5a25e308" exitCode=0
Sep 07 21:23:25.387277 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.387174    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x" event={"ID":"d39867b3-63b4-44ba-9fae-a7feb7be9cfe","Type":"ContainerDied","Data":"ad4913d6ead72bc78d30ee40e71e635dae929ecf63f6cf8871cae41d5a25e308"}
Sep 07 21:23:25.389175 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.389150    2565 generic.go:356] "Generic (PLEG): container finished" podID="974b1009-5faf-44cf-8d85-203ee50b85c4" containerID="72d24029f31f459f6baf80427be2661f61ca6c382c11ade860bf4783ac65e340" exitCode=0
Sep 07 21:23:25.389291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.389187    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg" event={"ID":"974b1009-5faf-44cf-8d85-203ee50b85c4","Type":"ContainerDied","Data":"72d24029f31f459f6baf80427be2661f61ca6c382c11ade860bf4783ac65e340"}
Sep 07 21:23:25.482824 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.482796    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:23:25.485794 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.485769    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:23:25.563750 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.563722    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974b1009-5faf-44cf-8d85-203ee50b85c4-plugins-config-volume\") pod \"974b1009-5faf-44cf-8d85-203ee50b85c4\" (UID: \"974b1009-5faf-44cf-8d85-203ee50b85c4\") "
Sep 07 21:23:25.563924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.563770    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l8jls\" (UniqueName: \"kubernetes.io/projected/974b1009-5faf-44cf-8d85-203ee50b85c4-kube-api-access-l8jls\") pod \"974b1009-5faf-44cf-8d85-203ee50b85c4\" (UID: \"974b1009-5faf-44cf-8d85-203ee50b85c4\") "
Sep 07 21:23:25.563924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.563791    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd5fm\" (UniqueName: \"kubernetes.io/projected/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-kube-api-access-xd5fm\") pod \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\" (UID: \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\") "
Sep 07 21:23:25.563924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.563853    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-plugins-config-volume\") pod \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\" (UID: \"d39867b3-63b4-44ba-9fae-a7feb7be9cfe\") "
Sep 07 21:23:25.564172 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.564135    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/974b1009-5faf-44cf-8d85-203ee50b85c4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "974b1009-5faf-44cf-8d85-203ee50b85c4" (UID: "974b1009-5faf-44cf-8d85-203ee50b85c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:23:25.564301 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.564279    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d39867b3-63b4-44ba-9fae-a7feb7be9cfe" (UID: "d39867b3-63b4-44ba-9fae-a7feb7be9cfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:23:25.565774 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.565752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-kube-api-access-xd5fm" (OuterVolumeSpecName: "kube-api-access-xd5fm") pod "d39867b3-63b4-44ba-9fae-a7feb7be9cfe" (UID: "d39867b3-63b4-44ba-9fae-a7feb7be9cfe"). InnerVolumeSpecName "kube-api-access-xd5fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:23:25.565835 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.565802    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/974b1009-5faf-44cf-8d85-203ee50b85c4-kube-api-access-l8jls" (OuterVolumeSpecName: "kube-api-access-l8jls") pod "974b1009-5faf-44cf-8d85-203ee50b85c4" (UID: "974b1009-5faf-44cf-8d85-203ee50b85c4"). InnerVolumeSpecName "kube-api-access-l8jls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:23:25.665315 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.665291    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8jls\" (UniqueName: \"kubernetes.io/projected/974b1009-5faf-44cf-8d85-203ee50b85c4-kube-api-access-l8jls\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:23:25.665315 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.665312    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd5fm\" (UniqueName: \"kubernetes.io/projected/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-kube-api-access-xd5fm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:23:25.665481 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.665323    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d39867b3-63b4-44ba-9fae-a7feb7be9cfe-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:23:25.665481 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:25.665331    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/974b1009-5faf-44cf-8d85-203ee50b85c4-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:23:26.364347 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.364306    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84947cf6-26ac-41df-b008-4a4e6c9e5af5" path="/var/lib/kubelet/pods/84947cf6-26ac-41df-b008-4a4e6c9e5af5/volumes"
Sep 07 21:23:26.395503 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.395481    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"
Sep 07 21:23:26.395643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.395508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg" event={"ID":"974b1009-5faf-44cf-8d85-203ee50b85c4","Type":"ContainerDied","Data":"a7babda1cd166beabe4db040415e7ec6df59345e71095b3043a7a29baa05a38b"}
Sep 07 21:23:26.395643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.395563    2565 scope.go:117] "RemoveContainer" containerID="72d24029f31f459f6baf80427be2661f61ca6c382c11ade860bf4783ac65e340"
Sep 07 21:23:26.396884 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.396857    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"
Sep 07 21:23:26.396957 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.396892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x" event={"ID":"d39867b3-63b4-44ba-9fae-a7feb7be9cfe","Type":"ContainerDied","Data":"13ceb9ee3c45c9a3b59f4dd9b4bb119d00a201d7cb2a8a546165a0d2e7c60cf4"}
Sep 07 21:23:26.406294 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.406275    2565 scope.go:117] "RemoveContainer" containerID="ad4913d6ead72bc78d30ee40e71e635dae929ecf63f6cf8871cae41d5a25e308"
Sep 07 21:23:26.422404 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.422380    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"]
Sep 07 21:23:26.425864 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.425841    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-424fbbc4-7c9774b889-wffjg"]
Sep 07 21:23:26.440019 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.439995    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"]
Sep 07 21:23:26.445461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:26.445438    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-424fbbc4-85cb4c8dc8-fg94x"]
Sep 07 21:23:28.364727 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:28.364690    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="974b1009-5faf-44cf-8d85-203ee50b85c4" path="/var/lib/kubelet/pods/974b1009-5faf-44cf-8d85-203ee50b85c4/volumes"
Sep 07 21:23:28.365256 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:28.365214    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d39867b3-63b4-44ba-9fae-a7feb7be9cfe" path="/var/lib/kubelet/pods/d39867b3-63b4-44ba-9fae-a7feb7be9cfe/volumes"
Sep 07 21:23:38.446044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:38.445996    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:23:48.572050 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:48.572018    2565 scope.go:117] "RemoveContainer" containerID="44b85ecaa3a3b884da00a479fe6bf10cb84e25bcaa65a2db563d353c5eaeb006"
Sep 07 21:23:48.580249 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:48.580229    2565 scope.go:117] "RemoveContainer" containerID="386b892a1a20c768ca61cc452791a955d7bd05a04aad99c5506b300bee00ca17"
Sep 07 21:23:55.751314 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:23:55.751281    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:24:14.162427 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:14.162347    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:24:32.364938 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:32.364903    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:24:33.834954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.834923    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7496-062e70"]
Sep 07 21:24:33.835349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835274    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84947cf6-26ac-41df-b008-4a4e6c9e5af5" containerName="revoke-keys"
Sep 07 21:24:33.835349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835289    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="84947cf6-26ac-41df-b008-4a4e6c9e5af5" containerName="revoke-keys"
Sep 07 21:24:33.835349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835305    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d39867b3-63b4-44ba-9fae-a7feb7be9cfe" containerName="payload-pre-processing"
Sep 07 21:24:33.835349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835310    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d39867b3-63b4-44ba-9fae-a7feb7be9cfe" containerName="payload-pre-processing"
Sep 07 21:24:33.835349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835317    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="974b1009-5faf-44cf-8d85-203ee50b85c4" containerName="payload-processing"
Sep 07 21:24:33.835349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835322    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="974b1009-5faf-44cf-8d85-203ee50b85c4" containerName="payload-processing"
Sep 07 21:24:33.835542 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835377    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="84947cf6-26ac-41df-b008-4a4e6c9e5af5" containerName="revoke-keys"
Sep 07 21:24:33.835542 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835386    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="974b1009-5faf-44cf-8d85-203ee50b85c4" containerName="payload-processing"
Sep 07 21:24:33.835542 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.835394    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d39867b3-63b4-44ba-9fae-a7feb7be9cfe" containerName="payload-pre-processing"
Sep 07 21:24:33.838451 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.838435    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:33.840660 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.840637    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6qlgr\""
Sep 07 21:24:33.844078 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.844053    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7496-062e70"]
Sep 07 21:24:33.960367 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:33.960330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdplc\" (UniqueName: \"kubernetes.io/projected/e676e618-bf92-499f-88e0-9c03c92029c5-kube-api-access-rdplc\") pod \"mt-curl-7496-062e70\" (UID: \"e676e618-bf92-499f-88e0-9c03c92029c5\") " pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:34.060846 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:34.060817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdplc\" (UniqueName: \"kubernetes.io/projected/e676e618-bf92-499f-88e0-9c03c92029c5-kube-api-access-rdplc\") pod \"mt-curl-7496-062e70\" (UID: \"e676e618-bf92-499f-88e0-9c03c92029c5\") " pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:34.068203 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:34.068176    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdplc\" (UniqueName: \"kubernetes.io/projected/e676e618-bf92-499f-88e0-9c03c92029c5-kube-api-access-rdplc\") pod \"mt-curl-7496-062e70\" (UID: \"e676e618-bf92-499f-88e0-9c03c92029c5\") " pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:34.148658 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:34.148588    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:34.414829 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:34.414804    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7496-062e70"]
Sep 07 21:24:34.417041 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:24:34.417012    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode676e618_bf92_499f_88e0_9c03c92029c5.slice/crio-44e905eb757acacf23b714b021d07ec06ad14bf265ae0e0fa7bf884a42c24ccb WatchSource:0}: Error finding container 44e905eb757acacf23b714b021d07ec06ad14bf265ae0e0fa7bf884a42c24ccb: Status 404 returned error can't find the container with id 44e905eb757acacf23b714b021d07ec06ad14bf265ae0e0fa7bf884a42c24ccb
Sep 07 21:24:34.724049 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:34.723947    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7496-062e70" event={"ID":"e676e618-bf92-499f-88e0-9c03c92029c5","Type":"ContainerStarted","Data":"44e905eb757acacf23b714b021d07ec06ad14bf265ae0e0fa7bf884a42c24ccb"}
Sep 07 21:24:35.730495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:35.730451    2565 generic.go:356] "Generic (PLEG): container finished" podID="e676e618-bf92-499f-88e0-9c03c92029c5" containerID="826b9f20165852d4c3f863de3be41009f95372b59b7062706dacfd4cc9ac94f7" exitCode=0
Sep 07 21:24:35.730900 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:35.730502    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7496-062e70" event={"ID":"e676e618-bf92-499f-88e0-9c03c92029c5","Type":"ContainerDied","Data":"826b9f20165852d4c3f863de3be41009f95372b59b7062706dacfd4cc9ac94f7"}
Sep 07 21:24:36.867589 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:36.867562    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:36.896512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:36.896491    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7496-062e70_e676e618-bf92-499f-88e0-9c03c92029c5/mt-curl-7496-062e70/0.log"
Sep 07 21:24:36.921394 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:36.921358    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7496-062e70"]
Sep 07 21:24:36.924817 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:36.924796    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7496-062e70"]
Sep 07 21:24:36.985896 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:36.985872    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rdplc\" (UniqueName: \"kubernetes.io/projected/e676e618-bf92-499f-88e0-9c03c92029c5-kube-api-access-rdplc\") pod \"e676e618-bf92-499f-88e0-9c03c92029c5\" (UID: \"e676e618-bf92-499f-88e0-9c03c92029c5\") "
Sep 07 21:24:36.987895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:36.987836    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e676e618-bf92-499f-88e0-9c03c92029c5-kube-api-access-rdplc" (OuterVolumeSpecName: "kube-api-access-rdplc") pod "e676e618-bf92-499f-88e0-9c03c92029c5" (UID: "e676e618-bf92-499f-88e0-9c03c92029c5"). InnerVolumeSpecName "kube-api-access-rdplc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:24:37.087466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:37.087430    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdplc\" (UniqueName: \"kubernetes.io/projected/e676e618-bf92-499f-88e0-9c03c92029c5-kube-api-access-rdplc\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:37.742355 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:37.742320    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:37.742537 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:37.742328    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44e905eb757acacf23b714b021d07ec06ad14bf265ae0e0fa7bf884a42c24ccb"
Sep 07 21:24:37.752027 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:24:37.752003    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7496-062e70\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="e676e618-bf92-499f-88e0-9c03c92029c5" pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:38.253684 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:38.253650    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"]
Sep 07 21:24:38.272527 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:24:38.272490    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7496-062e70\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="e676e618-bf92-499f-88e0-9c03c92029c5" pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:38.362075 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:24:38.362040    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7496-062e70\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="e676e618-bf92-499f-88e0-9c03c92029c5" pod="opendatahub/mt-curl-7496-062e70"
Sep 07 21:24:38.364522 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:38.364494    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e676e618-bf92-499f-88e0-9c03c92029c5" path="/var/lib/kubelet/pods/e676e618-bf92-499f-88e0-9c03c92029c5/volumes"
Sep 07 21:24:39.252157 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:39.252125    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"]
Sep 07 21:24:43.954736 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:43.954694    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:24:44.943570 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:44.943538    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"]
Sep 07 21:24:44.943898 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:44.943884    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e676e618-bf92-499f-88e0-9c03c92029c5" containerName="mt-curl-7496-062e70"
Sep 07 21:24:44.943941 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:44.943899    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e676e618-bf92-499f-88e0-9c03c92029c5" containerName="mt-curl-7496-062e70"
Sep 07 21:24:44.944008 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:44.943986    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e676e618-bf92-499f-88e0-9c03c92029c5" containerName="mt-curl-7496-062e70"
Sep 07 21:24:44.947086 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:44.947071    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:44.949520 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:44.949488    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"sub-test-model-15d5e2-kserve-self-signed-certs\""
Sep 07 21:24:44.954451 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:44.954431    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"]
Sep 07 21:24:45.058042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.058006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-dshm\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.058042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.058045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-tmp-dir\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.058484 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.058076    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83473f2b-8192-40d1-a7f5-b41d94f858ba-tls-certs\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.058484 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.058150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj8m2\" (UniqueName: \"kubernetes.io/projected/83473f2b-8192-40d1-a7f5-b41d94f858ba-kube-api-access-jj8m2\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.058484 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.058191    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-model-cache\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.058484 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.058240    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-home\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.159567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.159528    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83473f2b-8192-40d1-a7f5-b41d94f858ba-tls-certs\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.159814 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.159596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj8m2\" (UniqueName: \"kubernetes.io/projected/83473f2b-8192-40d1-a7f5-b41d94f858ba-kube-api-access-jj8m2\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.159814 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.159635    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-model-cache\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.159891 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.159827    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-home\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.159891 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.159871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-dshm\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.160007 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.159940    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-tmp-dir\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.160458 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.160416    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-model-cache\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.162991 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.160603    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-home\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.162991 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.160813    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-tmp-dir\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.167849 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.167819    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-dshm\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.168265 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.168243    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83473f2b-8192-40d1-a7f5-b41d94f858ba-tls-certs\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.170129 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.170108    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj8m2\" (UniqueName: \"kubernetes.io/projected/83473f2b-8192-40d1-a7f5-b41d94f858ba-kube-api-access-jj8m2\") pod \"sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.258574 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.258487    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:45.528293 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.528268    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"]
Sep 07 21:24:45.530721 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:24:45.530694    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83473f2b_8192_40d1_a7f5_b41d94f858ba.slice/crio-a75c343cf53337e6991ad2068a4b28880e9770f82248c20213ee7f5b995ab7bc WatchSource:0}: Error finding container a75c343cf53337e6991ad2068a4b28880e9770f82248c20213ee7f5b995ab7bc: Status 404 returned error can't find the container with id a75c343cf53337e6991ad2068a4b28880e9770f82248c20213ee7f5b995ab7bc
Sep 07 21:24:45.784202 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.784109    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn" event={"ID":"83473f2b-8192-40d1-a7f5-b41d94f858ba","Type":"ContainerStarted","Data":"aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa"}
Sep 07 21:24:45.784202 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.784152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn" event={"ID":"83473f2b-8192-40d1-a7f5-b41d94f858ba","Type":"ContainerStarted","Data":"a75c343cf53337e6991ad2068a4b28880e9770f82248c20213ee7f5b995ab7bc"}
Sep 07 21:24:45.799890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:45.799839    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn" podStartSLOduration=1.799820039 podStartE2EDuration="1.799820039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:24:45.799766936 +0000 UTC m=+1377.881294277" watchObservedRunningTime="2026-09-07 21:24:45.799820039 +0000 UTC m=+1377.881347380"
Sep 07 21:24:48.611349 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:48.611318    2565 scope.go:117] "RemoveContainer" containerID="17505dcd7cbe3ec8e345b4daf60cc52c669ba17bf25d05638062343631ae9f7a"
Sep 07 21:24:48.619594 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:48.619557    2565 scope.go:117] "RemoveContainer" containerID="6ca45a5d6a1a7bbd875321f805b09feae9ce09e518c14e01c098bcd120b9b8fd"
Sep 07 21:24:53.274122 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.274059    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l" podUID="0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" containerName="main" containerID="cri-o://f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4" gracePeriod=45
Sep 07 21:24:53.527717 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.527661    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:24:53.640627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.640597    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqmvn\" (UniqueName: \"kubernetes.io/projected/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-kube-api-access-sqmvn\") pod \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") "
Sep 07 21:24:53.640627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.640630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-dshm\") pod \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") "
Sep 07 21:24:53.640861 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.640666    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tmp-dir\") pod \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") "
Sep 07 21:24:53.640861 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.640779    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-home\") pod \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") "
Sep 07 21:24:53.640861 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.640850    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-model-cache\") pod \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") "
Sep 07 21:24:53.641041 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.640919    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tls-certs\") pod \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\" (UID: \"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b\") "
Sep 07 21:24:53.641041 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.640918    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" (UID: "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:24:53.641137 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.641053    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-home" (OuterVolumeSpecName: "home") pod "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" (UID: "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:24:53.641184 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.641126    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" (UID: "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:24:53.641333 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.641317    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:53.641413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.641334    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:53.641413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.641344    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:53.642915 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.642888    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-dshm" (OuterVolumeSpecName: "dshm") pod "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" (UID: "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:24:53.643035 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.642976    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" (UID: "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:24:53.643083 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.643045    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-kube-api-access-sqmvn" (OuterVolumeSpecName: "kube-api-access-sqmvn") pod "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" (UID: "0831cc86-e01d-42f2-8a70-4c8c76fe9f3b"). InnerVolumeSpecName "kube-api-access-sqmvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:24:53.742032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.741986    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:53.742032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.742035    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqmvn\" (UniqueName: \"kubernetes.io/projected/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-kube-api-access-sqmvn\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:53.742032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.742048    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:53.824151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.824061    2565 generic.go:356] "Generic (PLEG): container finished" podID="0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" containerID="f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4" exitCode=0
Sep 07 21:24:53.824151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.824111    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"
Sep 07 21:24:53.824365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.824151    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l" event={"ID":"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b","Type":"ContainerDied","Data":"f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4"}
Sep 07 21:24:53.824365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.824194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l" event={"ID":"0831cc86-e01d-42f2-8a70-4c8c76fe9f3b","Type":"ContainerDied","Data":"9bc1fd188e56e9224ddf13a4da5751b703fc50cf73b9ddbe219da48314ed13b9"}
Sep 07 21:24:53.824365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.824216    2565 scope.go:117] "RemoveContainer" containerID="f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4"
Sep 07 21:24:53.833888 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.833870    2565 scope.go:117] "RemoveContainer" containerID="f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4"
Sep 07 21:24:53.834233 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:24:53.834210    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4\": container with ID starting with f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4 not found: ID does not exist" containerID="f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4"
Sep 07 21:24:53.834287 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.834242    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4"} err="failed to get container status \"f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4\": rpc error: code = NotFound desc = could not find container \"f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4\": container with ID starting with f87b72be0575965e500b26b17547b2dcb49127b3ad07372524d90f6072b112d4 not found: ID does not exist"
Sep 07 21:24:53.844928 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.844907    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"]
Sep 07 21:24:53.846476 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:53.846447    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/auth-test-model-15d5e2-kserve-6d8d9699-gv75l"]
Sep 07 21:24:54.272072 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.272017    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr" podUID="e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" containerName="main" containerID="cri-o://0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be" gracePeriod=45
Sep 07 21:24:54.364291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.364262    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" path="/var/lib/kubelet/pods/0831cc86-e01d-42f2-8a70-4c8c76fe9f3b/volumes"
Sep 07 21:24:54.518621 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.518596    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:24:54.550116 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.550038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:24:54.649923 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.649893    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tmp-dir\") pod \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") "
Sep 07 21:24:54.649923 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.649927    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-dshm\") pod \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") "
Sep 07 21:24:54.650128 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tls-certs\") pod \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") "
Sep 07 21:24:54.650128 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650099    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-model-cache\") pod \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") "
Sep 07 21:24:54.650215 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650138    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" (UID: "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:24:54.650215 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-home\") pod \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") "
Sep 07 21:24:54.650215 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650198    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmjwv\" (UniqueName: \"kubernetes.io/projected/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-kube-api-access-xmjwv\") pod \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\" (UID: \"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed\") "
Sep 07 21:24:54.650420 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650400    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-home" (OuterVolumeSpecName: "home") pod "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" (UID: "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:24:54.650521 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650418    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-model-cache" (OuterVolumeSpecName: "model-cache") pod "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" (UID: "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:24:54.650582 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650553    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:54.650582 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650572    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:54.650669 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.650584    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:54.652738 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.652712    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" (UID: "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:24:54.652838 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.652752    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-dshm" (OuterVolumeSpecName: "dshm") pod "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" (UID: "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:24:54.652838 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.652765    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-kube-api-access-xmjwv" (OuterVolumeSpecName: "kube-api-access-xmjwv") pod "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" (UID: "e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed"). InnerVolumeSpecName "kube-api-access-xmjwv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:24:54.751360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.751330    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmjwv\" (UniqueName: \"kubernetes.io/projected/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-kube-api-access-xmjwv\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:54.751360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.751353    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:54.751525 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.751364    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:24:54.831545 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.831465    2565 generic.go:356] "Generic (PLEG): container finished" podID="e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" containerID="0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be" exitCode=0
Sep 07 21:24:54.831545 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.831511    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr" event={"ID":"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed","Type":"ContainerDied","Data":"0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be"}
Sep 07 21:24:54.831545 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.831536    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr" event={"ID":"e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed","Type":"ContainerDied","Data":"c3268dd0a22470340cb1d48340094d146b35fa0a9fa3e63c8162a078cf3ca4dd"}
Sep 07 21:24:54.831774 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.831531    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"
Sep 07 21:24:54.831774 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.831557    2565 scope.go:117] "RemoveContainer" containerID="0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be"
Sep 07 21:24:54.842370 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.842351    2565 scope.go:117] "RemoveContainer" containerID="0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be"
Sep 07 21:24:54.842740 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:24:54.842717    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be\": container with ID starting with 0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be not found: ID does not exist" containerID="0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be"
Sep 07 21:24:54.842817 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.842747    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be"} err="failed to get container status \"0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be\": rpc error: code = NotFound desc = could not find container \"0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be\": container with ID starting with 0ac55ffc0870ecb9f3f08ef3b98cfc7fb957ed3868f2c8f024569b59f59a65be not found: ID does not exist"
Sep 07 21:24:54.854097 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.854073    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"]
Sep 07 21:24:54.859936 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:54.859914    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/auth-test-model-7fa21f-kserve-54dfb64875-gthdr"]
Sep 07 21:24:55.259611 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:55.259572    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:55.259763 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:55.259631    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:55.272297 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:55.272268    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:55.849045 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:55.849016    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:24:56.364431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:24:56.364404    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" path="/var/lib/kubelet/pods/e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed/volumes"
Sep 07 21:25:08.459464 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.459429    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"]
Sep 07 21:25:08.459823 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.459766    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" containerName="main"
Sep 07 21:25:08.459823 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.459777    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" containerName="main"
Sep 07 21:25:08.459823 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.459792    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" containerName="main"
Sep 07 21:25:08.459823 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.459797    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" containerName="main"
Sep 07 21:25:08.459958 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.459861    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0831cc86-e01d-42f2-8a70-4c8c76fe9f3b" containerName="main"
Sep 07 21:25:08.459958 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.459869    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e90baf1a-2cd9-4501-b7fa-c13cd6a2f6ed" containerName="main"
Sep 07 21:25:08.465291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.465269    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.467878 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.467855    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"openshift-service-ca.crt\""
Sep 07 21:25:08.468047 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.467884    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"default-dockercfg-jpnn6\""
Sep 07 21:25:08.468776 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.468758    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"kube-root-ca.crt\""
Sep 07 21:25:08.468867 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.468853    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"sub-test-model-7fa21f-kserve-self-signed-certs\""
Sep 07 21:25:08.474470 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.474444    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"]
Sep 07 21:25:08.562831 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.562806    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-model-cache\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.562955 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.562840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-tmp-dir\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.562955 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.562855    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fe130f-b867-44e4-8ead-43ca802b74b5-tls-certs\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.563047 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.562982    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-dshm\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.563047 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.563008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-home\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.563117 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.563063    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkcb\" (UniqueName: \"kubernetes.io/projected/06fe130f-b867-44e4-8ead-43ca802b74b5-kube-api-access-6nkcb\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.663738 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.663708    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkcb\" (UniqueName: \"kubernetes.io/projected/06fe130f-b867-44e4-8ead-43ca802b74b5-kube-api-access-6nkcb\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.663882 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.663767    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-model-cache\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.663882 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.663787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-tmp-dir\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.663882 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.663802    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fe130f-b867-44e4-8ead-43ca802b74b5-tls-certs\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.663882 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.663842    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-dshm\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.664102 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.663911    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-home\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.664282 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.664257    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-tmp-dir\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.664343 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.664284    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-model-cache\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.664343 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.664326    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-home\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.666190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.666170    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-dshm\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.666332 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.666315    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fe130f-b867-44e4-8ead-43ca802b74b5-tls-certs\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.671044 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.671013    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkcb\" (UniqueName: \"kubernetes.io/projected/06fe130f-b867-44e4-8ead-43ca802b74b5-kube-api-access-6nkcb\") pod \"sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:08.775982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:08.775889    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:09.047905 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:09.047828    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"]
Sep 07 21:25:09.051266 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:25:09.051233    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06fe130f_b867_44e4_8ead_43ca802b74b5.slice/crio-887471b0f5ff6328056280920c943f0a26545e0988ee4ec45484d26420b4d14b WatchSource:0}: Error finding container 887471b0f5ff6328056280920c943f0a26545e0988ee4ec45484d26420b4d14b: Status 404 returned error can't find the container with id 887471b0f5ff6328056280920c943f0a26545e0988ee4ec45484d26420b4d14b
Sep 07 21:25:09.910137 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:09.910101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l" event={"ID":"06fe130f-b867-44e4-8ead-43ca802b74b5","Type":"ContainerStarted","Data":"b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232"}
Sep 07 21:25:09.910137 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:09.910141    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l" event={"ID":"06fe130f-b867-44e4-8ead-43ca802b74b5","Type":"ContainerStarted","Data":"887471b0f5ff6328056280920c943f0a26545e0988ee4ec45484d26420b4d14b"}
Sep 07 21:25:09.930727 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:09.930670    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l" podStartSLOduration=1.9306552959999999 podStartE2EDuration="1.930655296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:25:09.928349869 +0000 UTC m=+1402.009877224" watchObservedRunningTime="2026-09-07 21:25:09.930655296 +0000 UTC m=+1402.012182692"
Sep 07 21:25:14.366337 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:14.366294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:25:15.275158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.275118    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"]
Sep 07 21:25:15.279028 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.279004    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.281461 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.281442    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4cf126-dockercfg-xhhmp\""
Sep 07 21:25:15.281571 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.281510    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4cf126\""
Sep 07 21:25:15.287202 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.286811    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"]
Sep 07 21:25:15.320980 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.320940    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2f421-ae22-4b86-af64-d7d0d5316673-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k\" (UID: \"eaf2f421-ae22-4b86-af64-d7d0d5316673\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.321129 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.321107    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqj2g\" (UniqueName: \"kubernetes.io/projected/eaf2f421-ae22-4b86-af64-d7d0d5316673-kube-api-access-fqj2g\") pod \"payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k\" (UID: \"eaf2f421-ae22-4b86-af64-d7d0d5316673\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.422239 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.422208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqj2g\" (UniqueName: \"kubernetes.io/projected/eaf2f421-ae22-4b86-af64-d7d0d5316673-kube-api-access-fqj2g\") pod \"payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k\" (UID: \"eaf2f421-ae22-4b86-af64-d7d0d5316673\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.422569 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.422255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2f421-ae22-4b86-af64-d7d0d5316673-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k\" (UID: \"eaf2f421-ae22-4b86-af64-d7d0d5316673\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.422948 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.422930    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2f421-ae22-4b86-af64-d7d0d5316673-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k\" (UID: \"eaf2f421-ae22-4b86-af64-d7d0d5316673\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.429941 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.429914    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqj2g\" (UniqueName: \"kubernetes.io/projected/eaf2f421-ae22-4b86-af64-d7d0d5316673-kube-api-access-fqj2g\") pod \"payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k\" (UID: \"eaf2f421-ae22-4b86-af64-d7d0d5316673\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.590661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.590634    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.860361 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.860302    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"]
Sep 07 21:25:15.863359 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:25:15.863327    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf2f421_ae22_4b86_af64_d7d0d5316673.slice/crio-bbcecab8810919fa4d7cdd9179660cadbabece601a250ddffa1ad7a243aa203a WatchSource:0}: Error finding container bbcecab8810919fa4d7cdd9179660cadbabece601a250ddffa1ad7a243aa203a: Status 404 returned error can't find the container with id bbcecab8810919fa4d7cdd9179660cadbabece601a250ddffa1ad7a243aa203a
Sep 07 21:25:15.946137 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.946094    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k" event={"ID":"eaf2f421-ae22-4b86-af64-d7d0d5316673","Type":"ContainerStarted","Data":"2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a"}
Sep 07 21:25:15.946266 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.946158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k" event={"ID":"eaf2f421-ae22-4b86-af64-d7d0d5316673","Type":"ContainerStarted","Data":"bbcecab8810919fa4d7cdd9179660cadbabece601a250ddffa1ad7a243aa203a"}
Sep 07 21:25:15.946327 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.946281    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:15.962992 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:15.962910    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k" podStartSLOduration=0.962894283 podStartE2EDuration="962.894283ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:25:15.960474441 +0000 UTC m=+1408.042001783" watchObservedRunningTime="2026-09-07 21:25:15.962894283 +0000 UTC m=+1408.044421651"
Sep 07 21:25:18.777025 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:18.776988    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:18.777440 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:18.777043    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:18.789724 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:18.789693    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:18.972594 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:18.972568    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:25:26.953029 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:26.952998    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:25:30.249529 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:30.249493    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:25:45.363360 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:25:45.363322    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:26:00.289578 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.289541    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7496-ef19e2"]
Sep 07 21:26:00.294174 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.294158    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-ef19e2"
Sep 07 21:26:00.296950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.296928    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6qlgr\""
Sep 07 21:26:00.298503 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.298479    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7496-ef19e2"]
Sep 07 21:26:00.409243 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.409195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826bc\" (UniqueName: \"kubernetes.io/projected/b11a3f0f-f640-468f-a2ae-7b0ba61ab769-kube-api-access-826bc\") pod \"mt-curl-7496-ef19e2\" (UID: \"b11a3f0f-f640-468f-a2ae-7b0ba61ab769\") " pod="opendatahub/mt-curl-7496-ef19e2"
Sep 07 21:26:00.509614 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.509580    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826bc\" (UniqueName: \"kubernetes.io/projected/b11a3f0f-f640-468f-a2ae-7b0ba61ab769-kube-api-access-826bc\") pod \"mt-curl-7496-ef19e2\" (UID: \"b11a3f0f-f640-468f-a2ae-7b0ba61ab769\") " pod="opendatahub/mt-curl-7496-ef19e2"
Sep 07 21:26:00.517502 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.517475    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-826bc\" (UniqueName: \"kubernetes.io/projected/b11a3f0f-f640-468f-a2ae-7b0ba61ab769-kube-api-access-826bc\") pod \"mt-curl-7496-ef19e2\" (UID: \"b11a3f0f-f640-468f-a2ae-7b0ba61ab769\") " pod="opendatahub/mt-curl-7496-ef19e2"
Sep 07 21:26:00.605060 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.605038    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-ef19e2"
Sep 07 21:26:00.869151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:00.869122    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7496-ef19e2"]
Sep 07 21:26:00.871776 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:26:00.871747    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb11a3f0f_f640_468f_a2ae_7b0ba61ab769.slice/crio-1416de9e72dafbc998b798feb6d42bc9eb98e687c0dc882e0087efc8663e7484 WatchSource:0}: Error finding container 1416de9e72dafbc998b798feb6d42bc9eb98e687c0dc882e0087efc8663e7484: Status 404 returned error can't find the container with id 1416de9e72dafbc998b798feb6d42bc9eb98e687c0dc882e0087efc8663e7484
Sep 07 21:26:01.174736 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:01.174710    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7496-ef19e2" event={"ID":"b11a3f0f-f640-468f-a2ae-7b0ba61ab769","Type":"ContainerStarted","Data":"1416de9e72dafbc998b798feb6d42bc9eb98e687c0dc882e0087efc8663e7484"}
Sep 07 21:26:02.181504 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:02.181462    2565 generic.go:356] "Generic (PLEG): container finished" podID="b11a3f0f-f640-468f-a2ae-7b0ba61ab769" containerID="efab0cd7c633ba0086432278da3298404f90a194f4a391f3adb9591c2863cd28" exitCode=0
Sep 07 21:26:02.181892 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:02.181547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7496-ef19e2" event={"ID":"b11a3f0f-f640-468f-a2ae-7b0ba61ab769","Type":"ContainerDied","Data":"efab0cd7c633ba0086432278da3298404f90a194f4a391f3adb9591c2863cd28"}
Sep 07 21:26:03.316530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.316504    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-ef19e2"
Sep 07 21:26:03.337162 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.337137    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7496-ef19e2_b11a3f0f-f640-468f-a2ae-7b0ba61ab769/mt-curl-7496-ef19e2/0.log"
Sep 07 21:26:03.367159 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.365724    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7496-ef19e2"]
Sep 07 21:26:03.370031 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.369995    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7496-ef19e2"]
Sep 07 21:26:03.434239 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.434205    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-826bc\" (UniqueName: \"kubernetes.io/projected/b11a3f0f-f640-468f-a2ae-7b0ba61ab769-kube-api-access-826bc\") pod \"b11a3f0f-f640-468f-a2ae-7b0ba61ab769\" (UID: \"b11a3f0f-f640-468f-a2ae-7b0ba61ab769\") "
Sep 07 21:26:03.436491 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.436429    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11a3f0f-f640-468f-a2ae-7b0ba61ab769-kube-api-access-826bc" (OuterVolumeSpecName: "kube-api-access-826bc") pod "b11a3f0f-f640-468f-a2ae-7b0ba61ab769" (UID: "b11a3f0f-f640-468f-a2ae-7b0ba61ab769"). InnerVolumeSpecName "kube-api-access-826bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:26:03.483695 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.483662    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7496-1b5523"]
Sep 07 21:26:03.484076 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.484063    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b11a3f0f-f640-468f-a2ae-7b0ba61ab769" containerName="mt-curl-7496-ef19e2"
Sep 07 21:26:03.484139 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.484078    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b11a3f0f-f640-468f-a2ae-7b0ba61ab769" containerName="mt-curl-7496-ef19e2"
Sep 07 21:26:03.484174 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.484142    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b11a3f0f-f640-468f-a2ae-7b0ba61ab769" containerName="mt-curl-7496-ef19e2"
Sep 07 21:26:03.487137 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.487120    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:03.493430 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.493377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7496-1b5523"]
Sep 07 21:26:03.535299 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.535258    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84vh\" (UniqueName: \"kubernetes.io/projected/0851b081-b9ea-4555-9950-8a21c3e6c4c7-kube-api-access-x84vh\") pod \"mt-curl-7496-1b5523\" (UID: \"0851b081-b9ea-4555-9950-8a21c3e6c4c7\") " pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:03.535428 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.535317    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-826bc\" (UniqueName: \"kubernetes.io/projected/b11a3f0f-f640-468f-a2ae-7b0ba61ab769-kube-api-access-826bc\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:03.636752 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.636704    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84vh\" (UniqueName: \"kubernetes.io/projected/0851b081-b9ea-4555-9950-8a21c3e6c4c7-kube-api-access-x84vh\") pod \"mt-curl-7496-1b5523\" (UID: \"0851b081-b9ea-4555-9950-8a21c3e6c4c7\") " pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:03.644098 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.644067    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84vh\" (UniqueName: \"kubernetes.io/projected/0851b081-b9ea-4555-9950-8a21c3e6c4c7-kube-api-access-x84vh\") pod \"mt-curl-7496-1b5523\" (UID: \"0851b081-b9ea-4555-9950-8a21c3e6c4c7\") " pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:03.797984 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:03.797884    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:04.065065 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:04.065040    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7496-1b5523"]
Sep 07 21:26:04.067593 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:26:04.067560    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0851b081_b9ea_4555_9950_8a21c3e6c4c7.slice/crio-d6a6a358e1a3cafaa5f1b063f1e38a0cb37f2ed675458dde5d8e5f6e9219ceee WatchSource:0}: Error finding container d6a6a358e1a3cafaa5f1b063f1e38a0cb37f2ed675458dde5d8e5f6e9219ceee: Status 404 returned error can't find the container with id d6a6a358e1a3cafaa5f1b063f1e38a0cb37f2ed675458dde5d8e5f6e9219ceee
Sep 07 21:26:04.191733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:04.191702    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-ef19e2"
Sep 07 21:26:04.191897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:04.191738    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1416de9e72dafbc998b798feb6d42bc9eb98e687c0dc882e0087efc8663e7484"
Sep 07 21:26:04.192952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:04.192919    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7496-1b5523" event={"ID":"0851b081-b9ea-4555-9950-8a21c3e6c4c7","Type":"ContainerStarted","Data":"d6a6a358e1a3cafaa5f1b063f1e38a0cb37f2ed675458dde5d8e5f6e9219ceee"}
Sep 07 21:26:04.202095 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:26:04.202068    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7496-ef19e2\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="b11a3f0f-f640-468f-a2ae-7b0ba61ab769" pod="opendatahub/mt-curl-7496-ef19e2"
Sep 07 21:26:04.364945 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:04.364914    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b11a3f0f-f640-468f-a2ae-7b0ba61ab769" path="/var/lib/kubelet/pods/b11a3f0f-f640-468f-a2ae-7b0ba61ab769/volumes"
Sep 07 21:26:05.199619 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:05.199585    2565 generic.go:356] "Generic (PLEG): container finished" podID="0851b081-b9ea-4555-9950-8a21c3e6c4c7" containerID="5f4037ea89413882ecbeab1559817aeab271567d9eb613cc79bc04fe09e626b4" exitCode=0
Sep 07 21:26:05.199824 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:05.199677    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7496-1b5523" event={"ID":"0851b081-b9ea-4555-9950-8a21c3e6c4c7","Type":"ContainerDied","Data":"5f4037ea89413882ecbeab1559817aeab271567d9eb613cc79bc04fe09e626b4"}
Sep 07 21:26:06.338994 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:06.338954    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:06.357527 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:06.357471    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7496-1b5523_0851b081-b9ea-4555-9950-8a21c3e6c4c7/mt-curl-7496-1b5523/0.log"
Sep 07 21:26:06.380316 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:06.380290    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7496-1b5523"]
Sep 07 21:26:06.383846 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:06.383819    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7496-1b5523"]
Sep 07 21:26:06.463976 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:06.463867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x84vh\" (UniqueName: \"kubernetes.io/projected/0851b081-b9ea-4555-9950-8a21c3e6c4c7-kube-api-access-x84vh\") pod \"0851b081-b9ea-4555-9950-8a21c3e6c4c7\" (UID: \"0851b081-b9ea-4555-9950-8a21c3e6c4c7\") "
Sep 07 21:26:06.466005 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:06.465979    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0851b081-b9ea-4555-9950-8a21c3e6c4c7-kube-api-access-x84vh" (OuterVolumeSpecName: "kube-api-access-x84vh") pod "0851b081-b9ea-4555-9950-8a21c3e6c4c7" (UID: "0851b081-b9ea-4555-9950-8a21c3e6c4c7"). InnerVolumeSpecName "kube-api-access-x84vh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:26:06.565265 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:06.565229    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x84vh\" (UniqueName: \"kubernetes.io/projected/0851b081-b9ea-4555-9950-8a21c3e6c4c7-kube-api-access-x84vh\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:07.210952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:07.210924    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:07.211145 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:07.210984    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6a6a358e1a3cafaa5f1b063f1e38a0cb37f2ed675458dde5d8e5f6e9219ceee"
Sep 07 21:26:07.221118 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:26:07.221083    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7496-1b5523\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="0851b081-b9ea-4555-9950-8a21c3e6c4c7" pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:08.334097 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:08.334062    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"]
Sep 07 21:26:08.352247 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:26:08.352206    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7496-1b5523\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="0851b081-b9ea-4555-9950-8a21c3e6c4c7" pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:08.361591 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:26:08.361561    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7496-1b5523\" is forbidden: User \"system:node:ip-10-0-128-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-28.ec2.internal' and this object" podUID="0851b081-b9ea-4555-9950-8a21c3e6c4c7" pod="opendatahub/mt-curl-7496-1b5523"
Sep 07 21:26:08.363421 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:08.363397    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0851b081-b9ea-4555-9950-8a21c3e6c4c7" path="/var/lib/kubelet/pods/0851b081-b9ea-4555-9950-8a21c3e6c4c7/volumes"
Sep 07 21:26:08.656837 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:08.656749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:26:09.571917 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:09.571875    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"]
Sep 07 21:26:10.008424 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.008386    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"]
Sep 07 21:26:10.008794 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.008782    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0851b081-b9ea-4555-9950-8a21c3e6c4c7" containerName="mt-curl-7496-1b5523"
Sep 07 21:26:10.008841 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.008796    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0851b081-b9ea-4555-9950-8a21c3e6c4c7" containerName="mt-curl-7496-1b5523"
Sep 07 21:26:10.008874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.008857    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0851b081-b9ea-4555-9950-8a21c3e6c4c7" containerName="mt-curl-7496-1b5523"
Sep 07 21:26:10.012190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.012171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"
Sep 07 21:26:10.014394 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.014374    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:26:10.014512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.014374    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:26:10.019646 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.019410    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"]
Sep 07 21:26:10.098826 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.098785    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289d47be-338f-4a84-93d3-f9d44125b0ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l\" (UID: \"289d47be-338f-4a84-93d3-f9d44125b0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"
Sep 07 21:26:10.200156 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.200116    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289d47be-338f-4a84-93d3-f9d44125b0ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l\" (UID: \"289d47be-338f-4a84-93d3-f9d44125b0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"
Sep 07 21:26:10.200754 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.200734    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289d47be-338f-4a84-93d3-f9d44125b0ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l\" (UID: \"289d47be-338f-4a84-93d3-f9d44125b0ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"
Sep 07 21:26:10.324106 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.324016    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"
Sep 07 21:26:10.602100 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:10.602074    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"]
Sep 07 21:26:10.604234 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:26:10.604208    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289d47be_338f_4a84_93d3_f9d44125b0ba.slice/crio-559bc71132d7ce8e411206a1947127e9d77c60d887f5a4a36b832ee42ecf62e7 WatchSource:0}: Error finding container 559bc71132d7ce8e411206a1947127e9d77c60d887f5a4a36b832ee42ecf62e7: Status 404 returned error can't find the container with id 559bc71132d7ce8e411206a1947127e9d77c60d887f5a4a36b832ee42ecf62e7
Sep 07 21:26:11.233414 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:11.233307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l" event={"ID":"289d47be-338f-4a84-93d3-f9d44125b0ba","Type":"ContainerStarted","Data":"0f0af791326d91f4603877cc95fc2f8d662f9be30dbb4f2407c41517a97cb35f"}
Sep 07 21:26:11.233414 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:11.233373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l" event={"ID":"289d47be-338f-4a84-93d3-f9d44125b0ba","Type":"ContainerStarted","Data":"559bc71132d7ce8e411206a1947127e9d77c60d887f5a4a36b832ee42ecf62e7"}
Sep 07 21:26:11.247172 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:11.247130    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l" podStartSLOduration=2.247116916 podStartE2EDuration="2.247116916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:26:11.24651348 +0000 UTC m=+1463.328040833" watchObservedRunningTime="2026-09-07 21:26:11.247116916 +0000 UTC m=+1463.328644256"
Sep 07 21:26:12.239574 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:12.239527    2565 generic.go:356] "Generic (PLEG): container finished" podID="289d47be-338f-4a84-93d3-f9d44125b0ba" containerID="0f0af791326d91f4603877cc95fc2f8d662f9be30dbb4f2407c41517a97cb35f" exitCode=0
Sep 07 21:26:12.239951 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:12.239620    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l" event={"ID":"289d47be-338f-4a84-93d3-f9d44125b0ba","Type":"ContainerDied","Data":"0f0af791326d91f4603877cc95fc2f8d662f9be30dbb4f2407c41517a97cb35f"}
Sep 07 21:26:13.377002 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:13.376958    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"
Sep 07 21:26:13.426755 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:13.426732    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289d47be-338f-4a84-93d3-f9d44125b0ba-maas-api-service-ca\") pod \"289d47be-338f-4a84-93d3-f9d44125b0ba\" (UID: \"289d47be-338f-4a84-93d3-f9d44125b0ba\") "
Sep 07 21:26:13.427080 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:13.427051    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289d47be-338f-4a84-93d3-f9d44125b0ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "289d47be-338f-4a84-93d3-f9d44125b0ba" (UID: "289d47be-338f-4a84-93d3-f9d44125b0ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:26:13.527884 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:13.527798    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289d47be-338f-4a84-93d3-f9d44125b0ba-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:14.250786 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:14.250753    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"
Sep 07 21:26:14.250983 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:14.250788    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l" event={"ID":"289d47be-338f-4a84-93d3-f9d44125b0ba","Type":"ContainerDied","Data":"559bc71132d7ce8e411206a1947127e9d77c60d887f5a4a36b832ee42ecf62e7"}
Sep 07 21:26:14.250983 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:14.250837    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="559bc71132d7ce8e411206a1947127e9d77c60d887f5a4a36b832ee42ecf62e7"
Sep 07 21:26:23.354145 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.354103    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn" podUID="83473f2b-8192-40d1-a7f5-b41d94f858ba" containerName="main" containerID="cri-o://aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa" gracePeriod=45
Sep 07 21:26:23.605936 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.605878    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:26:23.724982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.724931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83473f2b-8192-40d1-a7f5-b41d94f858ba-tls-certs\") pod \"83473f2b-8192-40d1-a7f5-b41d94f858ba\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") "
Sep 07 21:26:23.725153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.724999    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj8m2\" (UniqueName: \"kubernetes.io/projected/83473f2b-8192-40d1-a7f5-b41d94f858ba-kube-api-access-jj8m2\") pod \"83473f2b-8192-40d1-a7f5-b41d94f858ba\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") "
Sep 07 21:26:23.725153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725026    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-model-cache\") pod \"83473f2b-8192-40d1-a7f5-b41d94f858ba\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") "
Sep 07 21:26:23.725153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-tmp-dir\") pod \"83473f2b-8192-40d1-a7f5-b41d94f858ba\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") "
Sep 07 21:26:23.725153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725115    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-home\") pod \"83473f2b-8192-40d1-a7f5-b41d94f858ba\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") "
Sep 07 21:26:23.725153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725132    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-dshm\") pod \"83473f2b-8192-40d1-a7f5-b41d94f858ba\" (UID: \"83473f2b-8192-40d1-a7f5-b41d94f858ba\") "
Sep 07 21:26:23.725415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725327    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-model-cache" (OuterVolumeSpecName: "model-cache") pod "83473f2b-8192-40d1-a7f5-b41d94f858ba" (UID: "83473f2b-8192-40d1-a7f5-b41d94f858ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:26:23.725415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725393    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "83473f2b-8192-40d1-a7f5-b41d94f858ba" (UID: "83473f2b-8192-40d1-a7f5-b41d94f858ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:26:23.725541 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725409    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-home" (OuterVolumeSpecName: "home") pod "83473f2b-8192-40d1-a7f5-b41d94f858ba" (UID: "83473f2b-8192-40d1-a7f5-b41d94f858ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:26:23.725541 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725465    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:23.725541 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725482    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:23.725541 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.725496    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:23.727162 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.727141    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-dshm" (OuterVolumeSpecName: "dshm") pod "83473f2b-8192-40d1-a7f5-b41d94f858ba" (UID: "83473f2b-8192-40d1-a7f5-b41d94f858ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:26:23.727288 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.727273    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83473f2b-8192-40d1-a7f5-b41d94f858ba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "83473f2b-8192-40d1-a7f5-b41d94f858ba" (UID: "83473f2b-8192-40d1-a7f5-b41d94f858ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:26:23.727330 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.727291    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83473f2b-8192-40d1-a7f5-b41d94f858ba-kube-api-access-jj8m2" (OuterVolumeSpecName: "kube-api-access-jj8m2") pod "83473f2b-8192-40d1-a7f5-b41d94f858ba" (UID: "83473f2b-8192-40d1-a7f5-b41d94f858ba"). InnerVolumeSpecName "kube-api-access-jj8m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:26:23.826906 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.826873    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83473f2b-8192-40d1-a7f5-b41d94f858ba-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:23.826906 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.826900    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj8m2\" (UniqueName: \"kubernetes.io/projected/83473f2b-8192-40d1-a7f5-b41d94f858ba-kube-api-access-jj8m2\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:23.826906 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:23.826909    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83473f2b-8192-40d1-a7f5-b41d94f858ba-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:24.148155 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.148123    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:26:24.300100 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.300062    2565 generic.go:356] "Generic (PLEG): container finished" podID="83473f2b-8192-40d1-a7f5-b41d94f858ba" containerID="aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa" exitCode=0
Sep 07 21:26:24.300297 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.300119    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn" event={"ID":"83473f2b-8192-40d1-a7f5-b41d94f858ba","Type":"ContainerDied","Data":"aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa"}
Sep 07 21:26:24.300297 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.300149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn" event={"ID":"83473f2b-8192-40d1-a7f5-b41d94f858ba","Type":"ContainerDied","Data":"a75c343cf53337e6991ad2068a4b28880e9770f82248c20213ee7f5b995ab7bc"}
Sep 07 21:26:24.300297 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.300165    2565 scope.go:117] "RemoveContainer" containerID="aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa"
Sep 07 21:26:24.300297 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.300121    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"
Sep 07 21:26:24.310260 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.310242    2565 scope.go:117] "RemoveContainer" containerID="aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa"
Sep 07 21:26:24.310524 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:26:24.310504    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa\": container with ID starting with aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa not found: ID does not exist" containerID="aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa"
Sep 07 21:26:24.310612 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.310531    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa"} err="failed to get container status \"aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa\": rpc error: code = NotFound desc = could not find container \"aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa\": container with ID starting with aaf64b23472cdb65e1fba2718c3559334b2216bf7b0d24b6843cbf3c284379fa not found: ID does not exist"
Sep 07 21:26:24.321560 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.321532    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"]
Sep 07 21:26:24.324975 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.324937    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/sub-test-model-15d5e2-kserve-7b89cb995f-fpkbn"]
Sep 07 21:26:24.364029 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.363945    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83473f2b-8192-40d1-a7f5-b41d94f858ba" path="/var/lib/kubelet/pods/83473f2b-8192-40d1-a7f5-b41d94f858ba/volumes"
Sep 07 21:26:24.591363 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.591324    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l" podUID="06fe130f-b867-44e4-8ead-43ca802b74b5" containerName="main" containerID="cri-o://b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232" gracePeriod=45
Sep 07 21:26:24.833356 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.833330    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:26:24.936644 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.936547    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-home\") pod \"06fe130f-b867-44e4-8ead-43ca802b74b5\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") "
Sep 07 21:26:24.936644 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.936607    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-tmp-dir\") pod \"06fe130f-b867-44e4-8ead-43ca802b74b5\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") "
Sep 07 21:26:24.936644 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.936637    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-dshm\") pod \"06fe130f-b867-44e4-8ead-43ca802b74b5\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") "
Sep 07 21:26:24.936932 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.936669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-model-cache\") pod \"06fe130f-b867-44e4-8ead-43ca802b74b5\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") "
Sep 07 21:26:24.936932 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.936688    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fe130f-b867-44e4-8ead-43ca802b74b5-tls-certs\") pod \"06fe130f-b867-44e4-8ead-43ca802b74b5\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") "
Sep 07 21:26:24.936932 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.936705    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nkcb\" (UniqueName: \"kubernetes.io/projected/06fe130f-b867-44e4-8ead-43ca802b74b5-kube-api-access-6nkcb\") pod \"06fe130f-b867-44e4-8ead-43ca802b74b5\" (UID: \"06fe130f-b867-44e4-8ead-43ca802b74b5\") "
Sep 07 21:26:24.937125 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.936857    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-home" (OuterVolumeSpecName: "home") pod "06fe130f-b867-44e4-8ead-43ca802b74b5" (UID: "06fe130f-b867-44e4-8ead-43ca802b74b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:26:24.937125 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.936917    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "06fe130f-b867-44e4-8ead-43ca802b74b5" (UID: "06fe130f-b867-44e4-8ead-43ca802b74b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:26:24.937125 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.937019    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-model-cache" (OuterVolumeSpecName: "model-cache") pod "06fe130f-b867-44e4-8ead-43ca802b74b5" (UID: "06fe130f-b867-44e4-8ead-43ca802b74b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:26:24.937244 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.937215    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:24.937244 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.937238    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:24.937316 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.937278    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:24.938852 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.938822    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-dshm" (OuterVolumeSpecName: "dshm") pod "06fe130f-b867-44e4-8ead-43ca802b74b5" (UID: "06fe130f-b867-44e4-8ead-43ca802b74b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:26:24.938955 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.938876    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06fe130f-b867-44e4-8ead-43ca802b74b5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "06fe130f-b867-44e4-8ead-43ca802b74b5" (UID: "06fe130f-b867-44e4-8ead-43ca802b74b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:26:24.938955 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:24.938881    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fe130f-b867-44e4-8ead-43ca802b74b5-kube-api-access-6nkcb" (OuterVolumeSpecName: "kube-api-access-6nkcb") pod "06fe130f-b867-44e4-8ead-43ca802b74b5" (UID: "06fe130f-b867-44e4-8ead-43ca802b74b5"). InnerVolumeSpecName "kube-api-access-6nkcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:26:25.038528 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.038487    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06fe130f-b867-44e4-8ead-43ca802b74b5-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:25.038528 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.038521    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06fe130f-b867-44e4-8ead-43ca802b74b5-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:25.038728 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.038537    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkcb\" (UniqueName: \"kubernetes.io/projected/06fe130f-b867-44e4-8ead-43ca802b74b5-kube-api-access-6nkcb\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:25.306215 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.306119    2565 generic.go:356] "Generic (PLEG): container finished" podID="06fe130f-b867-44e4-8ead-43ca802b74b5" containerID="b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232" exitCode=0
Sep 07 21:26:25.306215 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.306165    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"
Sep 07 21:26:25.306215 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.306168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l" event={"ID":"06fe130f-b867-44e4-8ead-43ca802b74b5","Type":"ContainerDied","Data":"b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232"}
Sep 07 21:26:25.306215 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.306215    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l" event={"ID":"06fe130f-b867-44e4-8ead-43ca802b74b5","Type":"ContainerDied","Data":"887471b0f5ff6328056280920c943f0a26545e0988ee4ec45484d26420b4d14b"}
Sep 07 21:26:25.306575 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.306233    2565 scope.go:117] "RemoveContainer" containerID="b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232"
Sep 07 21:26:25.315796 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.315778    2565 scope.go:117] "RemoveContainer" containerID="b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232"
Sep 07 21:26:25.316069 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:26:25.316048    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232\": container with ID starting with b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232 not found: ID does not exist" containerID="b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232"
Sep 07 21:26:25.316168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.316073    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232"} err="failed to get container status \"b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232\": rpc error: code = NotFound desc = could not find container \"b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232\": container with ID starting with b335684bccbc2036fc62cccd6a9bb819fe15a723239dde2bec7624ff28ec6232 not found: ID does not exist"
Sep 07 21:26:25.327119 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.327097    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"]
Sep 07 21:26:25.332797 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.332774    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/sub-test-model-7fa21f-kserve-7fb47c7f69-6fx5l"]
Sep 07 21:26:25.477004 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.474267    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"]
Sep 07 21:26:25.477004 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.474605    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k" podUID="eaf2f421-ae22-4b86-af64-d7d0d5316673" containerName="payload-processing" containerID="cri-o://2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a" gracePeriod=30
Sep 07 21:26:25.622806 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.622767    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"]
Sep 07 21:26:25.626051 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.626018    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cf126-xxn5l"]
Sep 07 21:26:25.734326 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.734301    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:26:25.846353 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.846325    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqj2g\" (UniqueName: \"kubernetes.io/projected/eaf2f421-ae22-4b86-af64-d7d0d5316673-kube-api-access-fqj2g\") pod \"eaf2f421-ae22-4b86-af64-d7d0d5316673\" (UID: \"eaf2f421-ae22-4b86-af64-d7d0d5316673\") "
Sep 07 21:26:25.846519 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.846394    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2f421-ae22-4b86-af64-d7d0d5316673-plugins-config-volume\") pod \"eaf2f421-ae22-4b86-af64-d7d0d5316673\" (UID: \"eaf2f421-ae22-4b86-af64-d7d0d5316673\") "
Sep 07 21:26:25.846758 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.846735    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf2f421-ae22-4b86-af64-d7d0d5316673-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eaf2f421-ae22-4b86-af64-d7d0d5316673" (UID: "eaf2f421-ae22-4b86-af64-d7d0d5316673"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:26:25.848418 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.848390    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf2f421-ae22-4b86-af64-d7d0d5316673-kube-api-access-fqj2g" (OuterVolumeSpecName: "kube-api-access-fqj2g") pod "eaf2f421-ae22-4b86-af64-d7d0d5316673" (UID: "eaf2f421-ae22-4b86-af64-d7d0d5316673"). InnerVolumeSpecName "kube-api-access-fqj2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:26:25.946974 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.946929    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqj2g\" (UniqueName: \"kubernetes.io/projected/eaf2f421-ae22-4b86-af64-d7d0d5316673-kube-api-access-fqj2g\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:25.946974 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:25.946959    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eaf2f421-ae22-4b86-af64-d7d0d5316673-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:26.314982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.314876    2565 generic.go:356] "Generic (PLEG): container finished" podID="eaf2f421-ae22-4b86-af64-d7d0d5316673" containerID="2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a" exitCode=0
Sep 07 21:26:26.314982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.314922    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"
Sep 07 21:26:26.314982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.314928    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k" event={"ID":"eaf2f421-ae22-4b86-af64-d7d0d5316673","Type":"ContainerDied","Data":"2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a"}
Sep 07 21:26:26.315240 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.314997    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k" event={"ID":"eaf2f421-ae22-4b86-af64-d7d0d5316673","Type":"ContainerDied","Data":"bbcecab8810919fa4d7cdd9179660cadbabece601a250ddffa1ad7a243aa203a"}
Sep 07 21:26:26.315240 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.315027    2565 scope.go:117] "RemoveContainer" containerID="2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a"
Sep 07 21:26:26.324326 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.324308    2565 scope.go:117] "RemoveContainer" containerID="2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a"
Sep 07 21:26:26.324620 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:26:26.324601    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a\": container with ID starting with 2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a not found: ID does not exist" containerID="2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a"
Sep 07 21:26:26.324674 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.324628    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a"} err="failed to get container status \"2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a\": rpc error: code = NotFound desc = could not find container \"2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a\": container with ID starting with 2ee625b8f5b03d44111d05416b92494cd0c9b452b8b3a9f8e19539a9f60e0a6a not found: ID does not exist"
Sep 07 21:26:26.336247 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.336224    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"]
Sep 07 21:26:26.340414 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.340391    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cf126-54bd9f7795-qqq8k"]
Sep 07 21:26:26.363894 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.363868    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06fe130f-b867-44e4-8ead-43ca802b74b5" path="/var/lib/kubelet/pods/06fe130f-b867-44e4-8ead-43ca802b74b5/volumes"
Sep 07 21:26:26.364257 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.364241    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289d47be-338f-4a84-93d3-f9d44125b0ba" path="/var/lib/kubelet/pods/289d47be-338f-4a84-93d3-f9d44125b0ba/volumes"
Sep 07 21:26:26.364566 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:26.364555    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaf2f421-ae22-4b86-af64-d7d0d5316673" path="/var/lib/kubelet/pods/eaf2f421-ae22-4b86-af64-d7d0d5316673/volumes"
Sep 07 21:26:32.505314 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505273    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"]
Sep 07 21:26:32.505719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505630    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83473f2b-8192-40d1-a7f5-b41d94f858ba" containerName="main"
Sep 07 21:26:32.505719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505642    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="83473f2b-8192-40d1-a7f5-b41d94f858ba" containerName="main"
Sep 07 21:26:32.505719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505650    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289d47be-338f-4a84-93d3-f9d44125b0ba" containerName="revoke-keys"
Sep 07 21:26:32.505719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505656    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="289d47be-338f-4a84-93d3-f9d44125b0ba" containerName="revoke-keys"
Sep 07 21:26:32.505719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505681    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06fe130f-b867-44e4-8ead-43ca802b74b5" containerName="main"
Sep 07 21:26:32.505719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505686    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="06fe130f-b867-44e4-8ead-43ca802b74b5" containerName="main"
Sep 07 21:26:32.505719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505693    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaf2f421-ae22-4b86-af64-d7d0d5316673" containerName="payload-processing"
Sep 07 21:26:32.505719 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505699    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaf2f421-ae22-4b86-af64-d7d0d5316673" containerName="payload-processing"
Sep 07 21:26:32.506000 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505753    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="289d47be-338f-4a84-93d3-f9d44125b0ba" containerName="revoke-keys"
Sep 07 21:26:32.506000 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505761    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="06fe130f-b867-44e4-8ead-43ca802b74b5" containerName="main"
Sep 07 21:26:32.506000 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505769    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="83473f2b-8192-40d1-a7f5-b41d94f858ba" containerName="main"
Sep 07 21:26:32.506000 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.505773    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaf2f421-ae22-4b86-af64-d7d0d5316673" containerName="payload-processing"
Sep 07 21:26:32.510217 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.510197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"
Sep 07 21:26:32.512811 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.512792    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:26:32.512934 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.512845    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:26:32.518310 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.518288    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"]
Sep 07 21:26:32.607358 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.607323    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d6380a-1543-4a52-a805-7c3f5f4258a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99\" (UID: \"b1d6380a-1543-4a52-a805-7c3f5f4258a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"
Sep 07 21:26:32.707740 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.707700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d6380a-1543-4a52-a805-7c3f5f4258a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99\" (UID: \"b1d6380a-1543-4a52-a805-7c3f5f4258a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"
Sep 07 21:26:32.708353 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.708332    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d6380a-1543-4a52-a805-7c3f5f4258a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99\" (UID: \"b1d6380a-1543-4a52-a805-7c3f5f4258a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"
Sep 07 21:26:32.820783 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:32.820689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"
Sep 07 21:26:33.092631 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:33.092606    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"]
Sep 07 21:26:33.353001 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:33.352901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99" event={"ID":"b1d6380a-1543-4a52-a805-7c3f5f4258a1","Type":"ContainerStarted","Data":"120e7fa668e572589f1189101fd515dccee81edaa6b29f6d4f5affcc35c65267"}
Sep 07 21:26:34.360590 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:34.360558    2565 generic.go:356] "Generic (PLEG): container finished" podID="b1d6380a-1543-4a52-a805-7c3f5f4258a1" containerID="498dadbc525479a3c9d821506874fbc619cae1b761c069690369ac77508b64a3" exitCode=0
Sep 07 21:26:34.363188 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:34.363163    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99" event={"ID":"b1d6380a-1543-4a52-a805-7c3f5f4258a1","Type":"ContainerDied","Data":"498dadbc525479a3c9d821506874fbc619cae1b761c069690369ac77508b64a3"}
Sep 07 21:26:35.493341 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:35.493314    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"
Sep 07 21:26:35.525407 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:35.525374    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d6380a-1543-4a52-a805-7c3f5f4258a1-maas-api-service-ca\") pod \"b1d6380a-1543-4a52-a805-7c3f5f4258a1\" (UID: \"b1d6380a-1543-4a52-a805-7c3f5f4258a1\") "
Sep 07 21:26:35.525722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:35.525699    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1d6380a-1543-4a52-a805-7c3f5f4258a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b1d6380a-1543-4a52-a805-7c3f5f4258a1" (UID: "b1d6380a-1543-4a52-a805-7c3f5f4258a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:26:35.626393 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:35.626310    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1d6380a-1543-4a52-a805-7c3f5f4258a1-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:36.372314 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:36.372289    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"
Sep 07 21:26:36.372314 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:36.372304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99" event={"ID":"b1d6380a-1543-4a52-a805-7c3f5f4258a1","Type":"ContainerDied","Data":"120e7fa668e572589f1189101fd515dccee81edaa6b29f6d4f5affcc35c65267"}
Sep 07 21:26:36.372477 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:36.372346    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="120e7fa668e572589f1189101fd515dccee81edaa6b29f6d4f5affcc35c65267"
Sep 07 21:26:37.350258 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.350229    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"]
Sep 07 21:26:37.350661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.350567    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1d6380a-1543-4a52-a805-7c3f5f4258a1" containerName="revoke-keys"
Sep 07 21:26:37.350661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.350578    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1d6380a-1543-4a52-a805-7c3f5f4258a1" containerName="revoke-keys"
Sep 07 21:26:37.350661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.350653    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1d6380a-1543-4a52-a805-7c3f5f4258a1" containerName="revoke-keys"
Sep 07 21:26:37.353682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.353664    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.356246 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.356224    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"kube-root-ca.crt\""
Sep 07 21:26:37.356365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.356251    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"openshift-service-ca.crt\""
Sep 07 21:26:37.357146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.357125    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"default-dockercfg-wbbfn\""
Sep 07 21:26:37.357270 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.357135    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-15d5e2\"/\"rate-test-model-15d5e2-kserve-self-signed-certs\""
Sep 07 21:26:37.364862 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.364842    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"]
Sep 07 21:26:37.438863 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.438825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-home\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.439032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.438875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tmp-dir\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.439032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.438915    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpzcn\" (UniqueName: \"kubernetes.io/projected/b9e03aec-911e-4b53-8a32-fc18c0c6751e-kube-api-access-zpzcn\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.439032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.439008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-dshm\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.439155 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.439040    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-model-cache\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.439155 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.439102    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tls-certs\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540329 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpzcn\" (UniqueName: \"kubernetes.io/projected/b9e03aec-911e-4b53-8a32-fc18c0c6751e-kube-api-access-zpzcn\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540329 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-dshm\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540346    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-model-cache\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540370    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tls-certs\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-home\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540554 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540518    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tmp-dir\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540833 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540811    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tmp-dir\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540833 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540825    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-model-cache\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.540948 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.540877    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-home\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.542518 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.542499    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-dshm\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.542842 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.542822    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tls-certs\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.551005 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.550952    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpzcn\" (UniqueName: \"kubernetes.io/projected/b9e03aec-911e-4b53-8a32-fc18c0c6751e-kube-api-access-zpzcn\") pod \"rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") " pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.656248 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.656167    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:26:37.663675 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.663656    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:37.933551 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:37.933521    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"]
Sep 07 21:26:37.935070 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:26:37.935048    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9e03aec_911e_4b53_8a32_fc18c0c6751e.slice/crio-889013670f75abe6183ed19b526f746e3e5692893ad146dd7b127eda971d8842 WatchSource:0}: Error finding container 889013670f75abe6183ed19b526f746e3e5692893ad146dd7b127eda971d8842: Status 404 returned error can't find the container with id 889013670f75abe6183ed19b526f746e3e5692893ad146dd7b127eda971d8842
Sep 07 21:26:38.385088 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:38.385051    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8" event={"ID":"b9e03aec-911e-4b53-8a32-fc18c0c6751e","Type":"ContainerStarted","Data":"e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a"}
Sep 07 21:26:38.385088 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:38.385094    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8" event={"ID":"b9e03aec-911e-4b53-8a32-fc18c0c6751e","Type":"ContainerStarted","Data":"889013670f75abe6183ed19b526f746e3e5692893ad146dd7b127eda971d8842"}
Sep 07 21:26:38.401575 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:38.401527    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8" podStartSLOduration=1.401511755 podStartE2EDuration="1.401511755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:26:38.400061055 +0000 UTC m=+1490.481588397" watchObservedRunningTime="2026-09-07 21:26:38.401511755 +0000 UTC m=+1490.483039096"
Sep 07 21:26:47.664668 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:47.664636    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:47.665120 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:47.664684    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:47.677354 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:47.677329    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:48.226407 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:48.226370    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"]
Sep 07 21:26:48.233982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:48.231004    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8f54f1-8xv99"]
Sep 07 21:26:48.364645 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:48.364611    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1d6380a-1543-4a52-a805-7c3f5f4258a1" path="/var/lib/kubelet/pods/b1d6380a-1543-4a52-a805-7c3f5f4258a1/volumes"
Sep 07 21:26:48.449585 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:48.447361    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:26:48.656823 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:48.656787    2565 scope.go:117] "RemoveContainer" containerID="a5dace73ca8c8da7c7a0fda1b755d420571469e65ef9b2e113c62084665c0973"
Sep 07 21:26:49.358865 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:49.358833    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:26:54.900253 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:54.900215    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"]
Sep 07 21:26:54.903926 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:54.903907    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"
Sep 07 21:26:54.906403 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:54.906378    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:26:54.906534 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:54.906506    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:26:54.911054 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:54.911026    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"]
Sep 07 21:26:54.996236 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:54.996198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/319adbc0-fc11-46cf-876f-39b6589b0e46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq\" (UID: \"319adbc0-fc11-46cf-876f-39b6589b0e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"
Sep 07 21:26:55.096794 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:55.096758    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/319adbc0-fc11-46cf-876f-39b6589b0e46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq\" (UID: \"319adbc0-fc11-46cf-876f-39b6589b0e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"
Sep 07 21:26:55.097414 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:55.097392    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/319adbc0-fc11-46cf-876f-39b6589b0e46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq\" (UID: \"319adbc0-fc11-46cf-876f-39b6589b0e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"
Sep 07 21:26:55.215333 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:55.215257    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"
Sep 07 21:26:55.483087 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:55.483062    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"]
Sep 07 21:26:55.485404 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:26:55.485366    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod319adbc0_fc11_46cf_876f_39b6589b0e46.slice/crio-80d393b50aca30c524819f076c1c972fb2bf01be4d2ecd736f68e31e7b2ef9d5 WatchSource:0}: Error finding container 80d393b50aca30c524819f076c1c972fb2bf01be4d2ecd736f68e31e7b2ef9d5: Status 404 returned error can't find the container with id 80d393b50aca30c524819f076c1c972fb2bf01be4d2ecd736f68e31e7b2ef9d5
Sep 07 21:26:56.470897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:56.470807    2565 generic.go:356] "Generic (PLEG): container finished" podID="319adbc0-fc11-46cf-876f-39b6589b0e46" containerID="698af8fba4c539de59b69a395bdd20a59693650e289163d14149910e45f6da8f" exitCode=0
Sep 07 21:26:56.470897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:56.470874    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq" event={"ID":"319adbc0-fc11-46cf-876f-39b6589b0e46","Type":"ContainerDied","Data":"698af8fba4c539de59b69a395bdd20a59693650e289163d14149910e45f6da8f"}
Sep 07 21:26:56.471359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:56.470910    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq" event={"ID":"319adbc0-fc11-46cf-876f-39b6589b0e46","Type":"ContainerStarted","Data":"80d393b50aca30c524819f076c1c972fb2bf01be4d2ecd736f68e31e7b2ef9d5"}
Sep 07 21:26:57.606364 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:57.606338    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"
Sep 07 21:26:57.719744 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:57.719712    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/319adbc0-fc11-46cf-876f-39b6589b0e46-maas-api-service-ca\") pod \"319adbc0-fc11-46cf-876f-39b6589b0e46\" (UID: \"319adbc0-fc11-46cf-876f-39b6589b0e46\") "
Sep 07 21:26:57.720098 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:57.720075    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/319adbc0-fc11-46cf-876f-39b6589b0e46-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "319adbc0-fc11-46cf-876f-39b6589b0e46" (UID: "319adbc0-fc11-46cf-876f-39b6589b0e46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:26:57.820951 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:57.820855    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/319adbc0-fc11-46cf-876f-39b6589b0e46-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:26:58.481479 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:58.481450    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"
Sep 07 21:26:58.481479 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:58.481464    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq" event={"ID":"319adbc0-fc11-46cf-876f-39b6589b0e46","Type":"ContainerDied","Data":"80d393b50aca30c524819f076c1c972fb2bf01be4d2ecd736f68e31e7b2ef9d5"}
Sep 07 21:26:58.481679 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:58.481509    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80d393b50aca30c524819f076c1c972fb2bf01be4d2ecd736f68e31e7b2ef9d5"
Sep 07 21:26:59.953641 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:26:59.953607    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:27:03.450741 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.450697    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"]
Sep 07 21:27:03.451290 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.451269    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="319adbc0-fc11-46cf-876f-39b6589b0e46" containerName="revoke-keys"
Sep 07 21:27:03.451379 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.451293    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="319adbc0-fc11-46cf-876f-39b6589b0e46" containerName="revoke-keys"
Sep 07 21:27:03.451439 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.451377    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="319adbc0-fc11-46cf-876f-39b6589b0e46" containerName="revoke-keys"
Sep 07 21:27:03.454463 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.454444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.458038 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.458004    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"openshift-service-ca.crt\""
Sep 07 21:27:03.458182 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.458160    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"default-dockercfg-jpnn6\""
Sep 07 21:27:03.458258 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.458164    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"kube-root-ca.crt\""
Sep 07 21:27:03.458417 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.458394    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7fa21f\"/\"rate-test-model-7fa21f-kserve-self-signed-certs\""
Sep 07 21:27:03.464369 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.464350    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"]
Sep 07 21:27:03.570955 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.570923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-dshm\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.571143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.571000    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rfft\" (UniqueName: \"kubernetes.io/projected/c6d3b184-e807-4439-aefd-ad342063c677-kube-api-access-8rfft\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.571143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.571025    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d3b184-e807-4439-aefd-ad342063c677-tls-certs\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.571143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.571049    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-home\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.571143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.571065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-model-cache\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.571143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.571132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-tmp-dir\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.671731 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.671700    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-model-cache\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.671899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.671744    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-tmp-dir\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.671899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.671785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-dshm\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.671899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.671829    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rfft\" (UniqueName: \"kubernetes.io/projected/c6d3b184-e807-4439-aefd-ad342063c677-kube-api-access-8rfft\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.671899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.671849    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d3b184-e807-4439-aefd-ad342063c677-tls-certs\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.671899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.671880    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-home\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.672198 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.672146    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-model-cache\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.672256 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.672197    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-tmp-dir\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.672318 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.672259    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-home\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.674138 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.674115    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-dshm\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.674496 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.674476    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d3b184-e807-4439-aefd-ad342063c677-tls-certs\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.679799 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.679781    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rfft\" (UniqueName: \"kubernetes.io/projected/c6d3b184-e807-4439-aefd-ad342063c677-kube-api-access-8rfft\") pod \"rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") " pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:03.764838 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:03.764763    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:04.036643 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:04.036617    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"]
Sep 07 21:27:04.038823 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:27:04.038795    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6d3b184_e807_4439_aefd_ad342063c677.slice/crio-26ec4d54ee388cb8bc71c480ded388eadd6b3f9537014280d3961b85caec8bf1 WatchSource:0}: Error finding container 26ec4d54ee388cb8bc71c480ded388eadd6b3f9537014280d3961b85caec8bf1: Status 404 returned error can't find the container with id 26ec4d54ee388cb8bc71c480ded388eadd6b3f9537014280d3961b85caec8bf1
Sep 07 21:27:04.513545 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:04.513508    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck" event={"ID":"c6d3b184-e807-4439-aefd-ad342063c677","Type":"ContainerStarted","Data":"30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2"}
Sep 07 21:27:04.513927 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:04.513569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck" event={"ID":"c6d3b184-e807-4439-aefd-ad342063c677","Type":"ContainerStarted","Data":"26ec4d54ee388cb8bc71c480ded388eadd6b3f9537014280d3961b85caec8bf1"}
Sep 07 21:27:04.529292 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:04.529246    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck" podStartSLOduration=1.5292304410000002 podStartE2EDuration="1.529230441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:27:04.527917696 +0000 UTC m=+1516.609445037" watchObservedRunningTime="2026-09-07 21:27:04.529230441 +0000 UTC m=+1516.610757781"
Sep 07 21:27:05.358671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.358634    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"]
Sep 07 21:27:05.359004 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.358972    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb" podUID="f7a44415-010b-4b3e-9057-a74bc15af9a7" containerName="payload-processing" containerID="cri-o://0ba9dac9deed910a13402106669c8d3068b7a6c184ec02977dd868b55159ed49" gracePeriod=30
Sep 07 21:27:05.511873 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.511839    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"]
Sep 07 21:27:05.515765 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.515734    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d03ead-q4jbq"]
Sep 07 21:27:05.520328 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.520299    2565 generic.go:356] "Generic (PLEG): container finished" podID="f7a44415-010b-4b3e-9057-a74bc15af9a7" containerID="0ba9dac9deed910a13402106669c8d3068b7a6c184ec02977dd868b55159ed49" exitCode=0
Sep 07 21:27:05.520429 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.520373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb" event={"ID":"f7a44415-010b-4b3e-9057-a74bc15af9a7","Type":"ContainerDied","Data":"0ba9dac9deed910a13402106669c8d3068b7a6c184ec02977dd868b55159ed49"}
Sep 07 21:27:05.607484 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.607460    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:27:05.691532 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.691441    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbw76\" (UniqueName: \"kubernetes.io/projected/f7a44415-010b-4b3e-9057-a74bc15af9a7-kube-api-access-hbw76\") pod \"f7a44415-010b-4b3e-9057-a74bc15af9a7\" (UID: \"f7a44415-010b-4b3e-9057-a74bc15af9a7\") "
Sep 07 21:27:05.691532 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.691478    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a44415-010b-4b3e-9057-a74bc15af9a7-plugins-config-volume\") pod \"f7a44415-010b-4b3e-9057-a74bc15af9a7\" (UID: \"f7a44415-010b-4b3e-9057-a74bc15af9a7\") "
Sep 07 21:27:05.691890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.691867    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7a44415-010b-4b3e-9057-a74bc15af9a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7a44415-010b-4b3e-9057-a74bc15af9a7" (UID: "f7a44415-010b-4b3e-9057-a74bc15af9a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:27:05.693753 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.693731    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7a44415-010b-4b3e-9057-a74bc15af9a7-kube-api-access-hbw76" (OuterVolumeSpecName: "kube-api-access-hbw76") pod "f7a44415-010b-4b3e-9057-a74bc15af9a7" (UID: "f7a44415-010b-4b3e-9057-a74bc15af9a7"). InnerVolumeSpecName "kube-api-access-hbw76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:27:05.792474 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.792445    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbw76\" (UniqueName: \"kubernetes.io/projected/f7a44415-010b-4b3e-9057-a74bc15af9a7-kube-api-access-hbw76\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:27:05.792474 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:05.792471    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7a44415-010b-4b3e-9057-a74bc15af9a7-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:27:06.363397 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:06.363359    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="319adbc0-fc11-46cf-876f-39b6589b0e46" path="/var/lib/kubelet/pods/319adbc0-fc11-46cf-876f-39b6589b0e46/volumes"
Sep 07 21:27:06.526413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:06.526387    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"
Sep 07 21:27:06.526413 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:06.526407    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb" event={"ID":"f7a44415-010b-4b3e-9057-a74bc15af9a7","Type":"ContainerDied","Data":"545eff32e78c4e09761e1a9fe05b97baea7f1808b8b709584c7173e235835b0c"}
Sep 07 21:27:06.526890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:06.526460    2565 scope.go:117] "RemoveContainer" containerID="0ba9dac9deed910a13402106669c8d3068b7a6c184ec02977dd868b55159ed49"
Sep 07 21:27:06.541640 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:06.541615    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"]
Sep 07 21:27:06.545153 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:06.545133    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d03ead-59ff9b79c5-qpdkb"]
Sep 07 21:27:08.364733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:08.364697    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7a44415-010b-4b3e-9057-a74bc15af9a7" path="/var/lib/kubelet/pods/f7a44415-010b-4b3e-9057-a74bc15af9a7/volumes"
Sep 07 21:27:12.106101 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.106016    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"]
Sep 07 21:27:12.106459 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.106354    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7a44415-010b-4b3e-9057-a74bc15af9a7" containerName="payload-processing"
Sep 07 21:27:12.106459 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.106365    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7a44415-010b-4b3e-9057-a74bc15af9a7" containerName="payload-processing"
Sep 07 21:27:12.106459 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.106439    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7a44415-010b-4b3e-9057-a74bc15af9a7" containerName="payload-processing"
Sep 07 21:27:12.110733 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.110716    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"
Sep 07 21:27:12.113239 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.113220    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:27:12.113356 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.113336    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:27:12.118429 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.118409    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"]
Sep 07 21:27:12.251071 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.251041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc54da4-50cc-4889-b896-c4672f81c9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m\" (UID: \"7fc54da4-50cc-4889-b896-c4672f81c9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"
Sep 07 21:27:12.351655 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.351626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc54da4-50cc-4889-b896-c4672f81c9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m\" (UID: \"7fc54da4-50cc-4889-b896-c4672f81c9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"
Sep 07 21:27:12.352251 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.352233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc54da4-50cc-4889-b896-c4672f81c9c2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m\" (UID: \"7fc54da4-50cc-4889-b896-c4672f81c9c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"
Sep 07 21:27:12.422723 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.422650    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"
Sep 07 21:27:12.693212 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:12.693060    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"]
Sep 07 21:27:12.695474 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:27:12.695441    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc54da4_50cc_4889_b896_c4672f81c9c2.slice/crio-85a126c547ec43e3958700ee7a0f7969ccd3b8702ecff1505e1a2092c500e04e WatchSource:0}: Error finding container 85a126c547ec43e3958700ee7a0f7969ccd3b8702ecff1505e1a2092c500e04e: Status 404 returned error can't find the container with id 85a126c547ec43e3958700ee7a0f7969ccd3b8702ecff1505e1a2092c500e04e
Sep 07 21:27:13.563208 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:13.563130    2565 generic.go:356] "Generic (PLEG): container finished" podID="7fc54da4-50cc-4889-b896-c4672f81c9c2" containerID="6cafd69d43edc470404ae88b2de74a976dbfeeaa991fffd61a1a729fa8528764" exitCode=0
Sep 07 21:27:13.563549 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:13.563213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m" event={"ID":"7fc54da4-50cc-4889-b896-c4672f81c9c2","Type":"ContainerDied","Data":"6cafd69d43edc470404ae88b2de74a976dbfeeaa991fffd61a1a729fa8528764"}
Sep 07 21:27:13.563549 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:13.563261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m" event={"ID":"7fc54da4-50cc-4889-b896-c4672f81c9c2","Type":"ContainerStarted","Data":"85a126c547ec43e3958700ee7a0f7969ccd3b8702ecff1505e1a2092c500e04e"}
Sep 07 21:27:13.765589 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:13.765556    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:13.765759 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:13.765639    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:13.778277 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:13.778255    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:14.580046 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:14.580019    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:27:14.711457 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:14.711436    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"
Sep 07 21:27:14.874706 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:14.874662    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc54da4-50cc-4889-b896-c4672f81c9c2-maas-api-service-ca\") pod \"7fc54da4-50cc-4889-b896-c4672f81c9c2\" (UID: \"7fc54da4-50cc-4889-b896-c4672f81c9c2\") "
Sep 07 21:27:14.875042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:14.875020    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc54da4-50cc-4889-b896-c4672f81c9c2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fc54da4-50cc-4889-b896-c4672f81c9c2" (UID: "7fc54da4-50cc-4889-b896-c4672f81c9c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:27:14.976246 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:14.976207    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fc54da4-50cc-4889-b896-c4672f81c9c2-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:27:15.574600 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:15.574571    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"
Sep 07 21:27:15.574772 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:15.574602    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m" event={"ID":"7fc54da4-50cc-4889-b896-c4672f81c9c2","Type":"ContainerDied","Data":"85a126c547ec43e3958700ee7a0f7969ccd3b8702ecff1505e1a2092c500e04e"}
Sep 07 21:27:15.574772 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:15.574653    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85a126c547ec43e3958700ee7a0f7969ccd3b8702ecff1505e1a2092c500e04e"
Sep 07 21:27:15.751031 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:15.750997    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:27:22.577631 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.577587    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"]
Sep 07 21:27:22.578080 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.577857    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2" podUID="8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" containerName="payload-processing" containerID="cri-o://3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3" gracePeriod=30
Sep 07 21:27:22.583879 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.583835    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"]
Sep 07 21:27:22.584165 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.584100    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9" podUID="4a1c3243-4dd2-41bb-b875-0654603152ac" containerName="payload-pre-processing" containerID="cri-o://ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c" gracePeriod=30
Sep 07 21:27:22.671220 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.671181    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9" podUID="4a1c3243-4dd2-41bb-b875-0654603152ac" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.60:9004: connect: connection refused"
Sep 07 21:27:22.671369 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.671198    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2" podUID="8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.61:9004: connect: connection refused"
Sep 07 21:27:22.732370 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.732334    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"]
Sep 07 21:27:22.734902 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.734877    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b9edc3-sw94m"]
Sep 07 21:27:22.865833 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.865807    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:27:22.868575 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.868554    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:27:22.944026 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.943991    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-plugins-config-volume\") pod \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\" (UID: \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\") "
Sep 07 21:27:22.944178 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.944044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dr658\" (UniqueName: \"kubernetes.io/projected/4a1c3243-4dd2-41bb-b875-0654603152ac-kube-api-access-dr658\") pod \"4a1c3243-4dd2-41bb-b875-0654603152ac\" (UID: \"4a1c3243-4dd2-41bb-b875-0654603152ac\") "
Sep 07 21:27:22.944178 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.944116    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1c3243-4dd2-41bb-b875-0654603152ac-plugins-config-volume\") pod \"4a1c3243-4dd2-41bb-b875-0654603152ac\" (UID: \"4a1c3243-4dd2-41bb-b875-0654603152ac\") "
Sep 07 21:27:22.944178 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.944164    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgtkk\" (UniqueName: \"kubernetes.io/projected/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-kube-api-access-vgtkk\") pod \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\" (UID: \"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2\") "
Sep 07 21:27:22.944505 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.944471    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" (UID: "8d42e4ee-9433-4bf7-9a70-c9281d6a81c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:27:22.944627 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.944511    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1c3243-4dd2-41bb-b875-0654603152ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a1c3243-4dd2-41bb-b875-0654603152ac" (UID: "4a1c3243-4dd2-41bb-b875-0654603152ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:27:22.946199 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.946176    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1c3243-4dd2-41bb-b875-0654603152ac-kube-api-access-dr658" (OuterVolumeSpecName: "kube-api-access-dr658") pod "4a1c3243-4dd2-41bb-b875-0654603152ac" (UID: "4a1c3243-4dd2-41bb-b875-0654603152ac"). InnerVolumeSpecName "kube-api-access-dr658". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:27:22.946301 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:22.946276    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-kube-api-access-vgtkk" (OuterVolumeSpecName: "kube-api-access-vgtkk") pod "8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" (UID: "8d42e4ee-9433-4bf7-9a70-c9281d6a81c2"). InnerVolumeSpecName "kube-api-access-vgtkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:27:23.044948 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.044916    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a1c3243-4dd2-41bb-b875-0654603152ac-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:27:23.044948 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.044940    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgtkk\" (UniqueName: \"kubernetes.io/projected/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-kube-api-access-vgtkk\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:27:23.044948 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.044949    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:27:23.045173 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.044959    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr658\" (UniqueName: \"kubernetes.io/projected/4a1c3243-4dd2-41bb-b875-0654603152ac-kube-api-access-dr658\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:27:23.617069 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.617034    2565 generic.go:356] "Generic (PLEG): container finished" podID="4a1c3243-4dd2-41bb-b875-0654603152ac" containerID="ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c" exitCode=0
Sep 07 21:27:23.617512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.617079    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"
Sep 07 21:27:23.617512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.617116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9" event={"ID":"4a1c3243-4dd2-41bb-b875-0654603152ac","Type":"ContainerDied","Data":"ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c"}
Sep 07 21:27:23.617512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.617166    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9" event={"ID":"4a1c3243-4dd2-41bb-b875-0654603152ac","Type":"ContainerDied","Data":"75010fe9879f3c8fedb12636e6d2524b7a8a850e9d37498d59c99f84075d6f88"}
Sep 07 21:27:23.617512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.617187    2565 scope.go:117] "RemoveContainer" containerID="ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c"
Sep 07 21:27:23.618573 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.618547    2565 generic.go:356] "Generic (PLEG): container finished" podID="8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" containerID="3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3" exitCode=0
Sep 07 21:27:23.618692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.618595    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"
Sep 07 21:27:23.618692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.618616    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2" event={"ID":"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2","Type":"ContainerDied","Data":"3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3"}
Sep 07 21:27:23.618692 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.618649    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2" event={"ID":"8d42e4ee-9433-4bf7-9a70-c9281d6a81c2","Type":"ContainerDied","Data":"73de537fa57a7d2a7dea6e4336633ad34862cac808e0c1f23549e17c3a0dc801"}
Sep 07 21:27:23.628207 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.628191    2565 scope.go:117] "RemoveContainer" containerID="ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c"
Sep 07 21:27:23.628571 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:27:23.628461    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c\": container with ID starting with ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c not found: ID does not exist" containerID="ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c"
Sep 07 21:27:23.628571 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.628481    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c"} err="failed to get container status \"ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c\": rpc error: code = NotFound desc = could not find container \"ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c\": container with ID starting with ee88e1498c51efab75a6fcf2d72db967a3ec04e65a00f18ff8f5ed71c0fb737c not found: ID does not exist"
Sep 07 21:27:23.628571 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.628504    2565 scope.go:117] "RemoveContainer" containerID="3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3"
Sep 07 21:27:23.636351 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.636332    2565 scope.go:117] "RemoveContainer" containerID="3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3"
Sep 07 21:27:23.636688 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:27:23.636662    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3\": container with ID starting with 3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3 not found: ID does not exist" containerID="3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3"
Sep 07 21:27:23.636867 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.636700    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3"} err="failed to get container status \"3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3\": rpc error: code = NotFound desc = could not find container \"3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3\": container with ID starting with 3ecda815806c01a77d9b3c39904626a76042f52588241e8f3edfaa7b52524fa3 not found: ID does not exist"
Sep 07 21:27:23.639084 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.639061    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"]
Sep 07 21:27:23.642661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.642642    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b9edc3-d8dc7474b-zglw9"]
Sep 07 21:27:23.653078 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.653036    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"]
Sep 07 21:27:23.655042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:23.655016    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b9edc3-654fcbd755-4nwv2"]
Sep 07 21:27:24.363431 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:24.363396    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1c3243-4dd2-41bb-b875-0654603152ac" path="/var/lib/kubelet/pods/4a1c3243-4dd2-41bb-b875-0654603152ac/volumes"
Sep 07 21:27:24.363732 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:24.363719    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fc54da4-50cc-4889-b896-c4672f81c9c2" path="/var/lib/kubelet/pods/7fc54da4-50cc-4889-b896-c4672f81c9c2/volumes"
Sep 07 21:27:24.364055 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:24.364041    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" path="/var/lib/kubelet/pods/8d42e4ee-9433-4bf7-9a70-c9281d6a81c2/volumes"
Sep 07 21:27:29.851916 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:29.851873    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:27:39.562763 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:39.562732    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:27:45.025252 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:45.025214    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"]
Sep 07 21:27:46.039437 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:46.039404    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"]
Sep 07 21:27:48.689489 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:48.689454    2565 scope.go:117] "RemoveContainer" containerID="0753f11629546b19d365efe3a3162389561d6297ededaec36ce09ddb46132390"
Sep 07 21:27:48.697759 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:48.697741    2565 scope.go:117] "RemoveContainer" containerID="ed558d6cb6eeace3e6d91c87f77f051215ca48f88970734b706bd0720d5f78f0"
Sep 07 21:27:50.659147 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:50.659113    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:27:52.066269 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.066233    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"]
Sep 07 21:27:52.066865 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.066844    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" containerName="payload-processing"
Sep 07 21:27:52.066923 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.066870    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" containerName="payload-processing"
Sep 07 21:27:52.066923 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.066899    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1c3243-4dd2-41bb-b875-0654603152ac" containerName="payload-pre-processing"
Sep 07 21:27:52.066923 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.066908    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1c3243-4dd2-41bb-b875-0654603152ac" containerName="payload-pre-processing"
Sep 07 21:27:52.067063 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.066932    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fc54da4-50cc-4889-b896-c4672f81c9c2" containerName="revoke-keys"
Sep 07 21:27:52.067063 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.066938    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fc54da4-50cc-4889-b896-c4672f81c9c2" containerName="revoke-keys"
Sep 07 21:27:52.067063 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.067024    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fc54da4-50cc-4889-b896-c4672f81c9c2" containerName="revoke-keys"
Sep 07 21:27:52.067063 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.067033    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d42e4ee-9433-4bf7-9a70-c9281d6a81c2" containerName="payload-processing"
Sep 07 21:27:52.067063 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.067041    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1c3243-4dd2-41bb-b875-0654603152ac" containerName="payload-pre-processing"
Sep 07 21:27:52.069839 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.069823    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.073020 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.072999    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-52269e\""
Sep 07 21:27:52.073143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.073002    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-52269e-dockercfg-jvtg5\""
Sep 07 21:27:52.083118 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.083093    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"]
Sep 07 21:27:52.195169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.195134    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f\" (UID: \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.195169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.195174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgtht\" (UniqueName: \"kubernetes.io/projected/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-kube-api-access-wgtht\") pod \"payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f\" (UID: \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.295931 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.295901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f\" (UID: \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.296086 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.295937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgtht\" (UniqueName: \"kubernetes.io/projected/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-kube-api-access-wgtht\") pod \"payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f\" (UID: \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.296578 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.296557    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f\" (UID: \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.303911 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.303884    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgtht\" (UniqueName: \"kubernetes.io/projected/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-kube-api-access-wgtht\") pod \"payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f\" (UID: \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.379671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.379645    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.647350 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.647329    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"]
Sep 07 21:27:52.649410 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:27:52.649383    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ab5ee1_6a4f_439a_b0cf_d1131612419e.slice/crio-e05b5aec050d7f8ed71c940924ba6faeed3e46885a715c72f3b4662f26a215ae WatchSource:0}: Error finding container e05b5aec050d7f8ed71c940924ba6faeed3e46885a715c72f3b4662f26a215ae: Status 404 returned error can't find the container with id e05b5aec050d7f8ed71c940924ba6faeed3e46885a715c72f3b4662f26a215ae
Sep 07 21:27:52.776983 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.776938    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f" event={"ID":"b2ab5ee1-6a4f-439a-b0cf-d1131612419e","Type":"ContainerStarted","Data":"ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f"}
Sep 07 21:27:52.777158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.777017    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f" event={"ID":"b2ab5ee1-6a4f-439a-b0cf-d1131612419e","Type":"ContainerStarted","Data":"e05b5aec050d7f8ed71c940924ba6faeed3e46885a715c72f3b4662f26a215ae"}
Sep 07 21:27:52.777158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.777088    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:27:52.792818 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:27:52.792755    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f" podStartSLOduration=0.792736723 podStartE2EDuration="792.736723ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:27:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:27:52.792429529 +0000 UTC m=+1564.873956870" watchObservedRunningTime="2026-09-07 21:27:52.792736723 +0000 UTC m=+1564.874264065"
Sep 07 21:28:00.044958 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.044921    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8" podUID="b9e03aec-911e-4b53-8a32-fc18c0c6751e" containerName="main" containerID="cri-o://e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a" gracePeriod=45
Sep 07 21:28:00.288807 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.288783    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:28:00.365197 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365168    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tmp-dir\") pod \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") "
Sep 07 21:28:00.365365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365262    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-home\") pod \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") "
Sep 07 21:28:00.365365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365284    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpzcn\" (UniqueName: \"kubernetes.io/projected/b9e03aec-911e-4b53-8a32-fc18c0c6751e-kube-api-access-zpzcn\") pod \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") "
Sep 07 21:28:00.365365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365302    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tls-certs\") pod \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") "
Sep 07 21:28:00.365365 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365328    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-dshm\") pod \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") "
Sep 07 21:28:00.365613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365437    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9e03aec-911e-4b53-8a32-fc18c0c6751e" (UID: "b9e03aec-911e-4b53-8a32-fc18c0c6751e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:28:00.365613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365532    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-model-cache\") pod \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\" (UID: \"b9e03aec-911e-4b53-8a32-fc18c0c6751e\") "
Sep 07 21:28:00.365613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365545    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-home" (OuterVolumeSpecName: "home") pod "b9e03aec-911e-4b53-8a32-fc18c0c6751e" (UID: "b9e03aec-911e-4b53-8a32-fc18c0c6751e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:28:00.365777 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365736    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9e03aec-911e-4b53-8a32-fc18c0c6751e" (UID: "b9e03aec-911e-4b53-8a32-fc18c0c6751e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:28:00.365852 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365830    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:00.365852 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365848    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:00.365956 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.365860    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:00.367591 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.367569    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-dshm" (OuterVolumeSpecName: "dshm") pod "b9e03aec-911e-4b53-8a32-fc18c0c6751e" (UID: "b9e03aec-911e-4b53-8a32-fc18c0c6751e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:28:00.367694 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.367604    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9e03aec-911e-4b53-8a32-fc18c0c6751e" (UID: "b9e03aec-911e-4b53-8a32-fc18c0c6751e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:28:00.367694 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.367623    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e03aec-911e-4b53-8a32-fc18c0c6751e-kube-api-access-zpzcn" (OuterVolumeSpecName: "kube-api-access-zpzcn") pod "b9e03aec-911e-4b53-8a32-fc18c0c6751e" (UID: "b9e03aec-911e-4b53-8a32-fc18c0c6751e"). InnerVolumeSpecName "kube-api-access-zpzcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:28:00.467168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.467133    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpzcn\" (UniqueName: \"kubernetes.io/projected/b9e03aec-911e-4b53-8a32-fc18c0c6751e-kube-api-access-zpzcn\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:00.467168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.467161    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9e03aec-911e-4b53-8a32-fc18c0c6751e-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:00.467168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.467176    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9e03aec-911e-4b53-8a32-fc18c0c6751e-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:00.823384 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.823289    2565 generic.go:356] "Generic (PLEG): container finished" podID="b9e03aec-911e-4b53-8a32-fc18c0c6751e" containerID="e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a" exitCode=0
Sep 07 21:28:00.823384 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.823331    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"
Sep 07 21:28:00.823722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.823378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8" event={"ID":"b9e03aec-911e-4b53-8a32-fc18c0c6751e","Type":"ContainerDied","Data":"e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a"}
Sep 07 21:28:00.823722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.823428    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8" event={"ID":"b9e03aec-911e-4b53-8a32-fc18c0c6751e","Type":"ContainerDied","Data":"889013670f75abe6183ed19b526f746e3e5692893ad146dd7b127eda971d8842"}
Sep 07 21:28:00.823722 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.823453    2565 scope.go:117] "RemoveContainer" containerID="e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a"
Sep 07 21:28:00.832657 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.832643    2565 scope.go:117] "RemoveContainer" containerID="e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a"
Sep 07 21:28:00.832922 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:28:00.832904    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a\": container with ID starting with e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a not found: ID does not exist" containerID="e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a"
Sep 07 21:28:00.833001 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.832931    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a"} err="failed to get container status \"e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a\": rpc error: code = NotFound desc = could not find container \"e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a\": container with ID starting with e38e745fe79688d8f53f0197a737dd4ea3c085a790a2617ea3dfda1e3c79b75a not found: ID does not exist"
Sep 07 21:28:00.843443 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.843421    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"]
Sep 07 21:28:00.845751 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:00.845731    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-15d5e2/rate-test-model-15d5e2-kserve-7b9c9bb4b7-8pcn8"]
Sep 07 21:28:01.059085 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.059043    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck" podUID="c6d3b184-e807-4439-aefd-ad342063c677" containerName="main" containerID="cri-o://30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2" gracePeriod=45
Sep 07 21:28:01.311605 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.311575    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:28:01.476022 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.475988    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-model-cache\") pod \"c6d3b184-e807-4439-aefd-ad342063c677\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") "
Sep 07 21:28:01.476195 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476042    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d3b184-e807-4439-aefd-ad342063c677-tls-certs\") pod \"c6d3b184-e807-4439-aefd-ad342063c677\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") "
Sep 07 21:28:01.476195 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476089    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8rfft\" (UniqueName: \"kubernetes.io/projected/c6d3b184-e807-4439-aefd-ad342063c677-kube-api-access-8rfft\") pod \"c6d3b184-e807-4439-aefd-ad342063c677\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") "
Sep 07 21:28:01.476195 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476118    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-home\") pod \"c6d3b184-e807-4439-aefd-ad342063c677\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") "
Sep 07 21:28:01.476195 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476144    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-tmp-dir\") pod \"c6d3b184-e807-4439-aefd-ad342063c677\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") "
Sep 07 21:28:01.476424 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476203    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-dshm\") pod \"c6d3b184-e807-4439-aefd-ad342063c677\" (UID: \"c6d3b184-e807-4439-aefd-ad342063c677\") "
Sep 07 21:28:01.476424 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476302    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6d3b184-e807-4439-aefd-ad342063c677" (UID: "c6d3b184-e807-4439-aefd-ad342063c677"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:28:01.476528 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476397    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-home" (OuterVolumeSpecName: "home") pod "c6d3b184-e807-4439-aefd-ad342063c677" (UID: "c6d3b184-e807-4439-aefd-ad342063c677"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:28:01.476528 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476465    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6d3b184-e807-4439-aefd-ad342063c677" (UID: "c6d3b184-e807-4439-aefd-ad342063c677"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:28:01.476528 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476493    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:01.476528 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.476507    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:01.478293 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.478268    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-dshm" (OuterVolumeSpecName: "dshm") pod "c6d3b184-e807-4439-aefd-ad342063c677" (UID: "c6d3b184-e807-4439-aefd-ad342063c677"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:28:01.478420 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.478297    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d3b184-e807-4439-aefd-ad342063c677-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6d3b184-e807-4439-aefd-ad342063c677" (UID: "c6d3b184-e807-4439-aefd-ad342063c677"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:28:01.478484 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.478463    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d3b184-e807-4439-aefd-ad342063c677-kube-api-access-8rfft" (OuterVolumeSpecName: "kube-api-access-8rfft") pod "c6d3b184-e807-4439-aefd-ad342063c677" (UID: "c6d3b184-e807-4439-aefd-ad342063c677"). InnerVolumeSpecName "kube-api-access-8rfft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:28:01.577362 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.577332    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d3b184-e807-4439-aefd-ad342063c677-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:01.577362 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.577357    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rfft\" (UniqueName: \"kubernetes.io/projected/c6d3b184-e807-4439-aefd-ad342063c677-kube-api-access-8rfft\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:01.577362 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.577367    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:01.577575 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.577382    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d3b184-e807-4439-aefd-ad342063c677-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:28:01.829918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.829828    2565 generic.go:356] "Generic (PLEG): container finished" podID="c6d3b184-e807-4439-aefd-ad342063c677" containerID="30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2" exitCode=0
Sep 07 21:28:01.829918 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.829869    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"
Sep 07 21:28:01.830171 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.829918    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck" event={"ID":"c6d3b184-e807-4439-aefd-ad342063c677","Type":"ContainerDied","Data":"30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2"}
Sep 07 21:28:01.830171 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.829957    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck" event={"ID":"c6d3b184-e807-4439-aefd-ad342063c677","Type":"ContainerDied","Data":"26ec4d54ee388cb8bc71c480ded388eadd6b3f9537014280d3961b85caec8bf1"}
Sep 07 21:28:01.830171 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.829999    2565 scope.go:117] "RemoveContainer" containerID="30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2"
Sep 07 21:28:01.838899 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.838875    2565 scope.go:117] "RemoveContainer" containerID="30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2"
Sep 07 21:28:01.839183 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:28:01.839160    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2\": container with ID starting with 30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2 not found: ID does not exist" containerID="30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2"
Sep 07 21:28:01.839238 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.839193    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2"} err="failed to get container status \"30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2\": rpc error: code = NotFound desc = could not find container \"30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2\": container with ID starting with 30772d71f415d17c2ada9c83274aeb72d035344b14b3bb55bf7007cc5dd551f2 not found: ID does not exist"
Sep 07 21:28:01.851355 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.851328    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"]
Sep 07 21:28:01.854544 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:01.854525    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7fa21f/rate-test-model-7fa21f-kserve-8494fdcf9f-9gfck"]
Sep 07 21:28:02.364794 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:02.364754    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9e03aec-911e-4b53-8a32-fc18c0c6751e" path="/var/lib/kubelet/pods/b9e03aec-911e-4b53-8a32-fc18c0c6751e/volumes"
Sep 07 21:28:02.365353 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:02.365331    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6d3b184-e807-4439-aefd-ad342063c677" path="/var/lib/kubelet/pods/c6d3b184-e807-4439-aefd-ad342063c677/volumes"
Sep 07 21:28:03.071353 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:03.071316    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:28:03.784900 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:03.784792    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:28:20.990939 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.990905    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"]
Sep 07 21:28:20.991412 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.991287    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9e03aec-911e-4b53-8a32-fc18c0c6751e" containerName="main"
Sep 07 21:28:20.991412 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.991298    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9e03aec-911e-4b53-8a32-fc18c0c6751e" containerName="main"
Sep 07 21:28:20.991412 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.991318    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6d3b184-e807-4439-aefd-ad342063c677" containerName="main"
Sep 07 21:28:20.991412 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.991323    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6d3b184-e807-4439-aefd-ad342063c677" containerName="main"
Sep 07 21:28:20.991412 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.991380    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6d3b184-e807-4439-aefd-ad342063c677" containerName="main"
Sep 07 21:28:20.991412 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.991388    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9e03aec-911e-4b53-8a32-fc18c0c6751e" containerName="main"
Sep 07 21:28:20.994150 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.994132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:20.996751 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.996720    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-629e95-dockercfg-hrcth\""
Sep 07 21:28:20.996888 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:20.996756    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-629e95\""
Sep 07 21:28:21.005694 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.005666    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"]
Sep 07 21:28:21.021261 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.021237    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd379c7-2619-4c19-b4e1-935d438fd8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq\" (UID: \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:21.021386 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.021282    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bs6\" (UniqueName: \"kubernetes.io/projected/6dd379c7-2619-4c19-b4e1-935d438fd8a6-kube-api-access-s7bs6\") pod \"payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq\" (UID: \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:21.024057 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.024034    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"]
Sep 07 21:28:21.027381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.027366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.036222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.036199    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"]
Sep 07 21:28:21.122503 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.122461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5911ad76-b763-4a9b-8134-206a6da66eb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66\" (UID: \"5911ad76-b763-4a9b-8134-206a6da66eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.122678 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.122513    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd379c7-2619-4c19-b4e1-935d438fd8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq\" (UID: \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:21.122678 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.122542    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bs6\" (UniqueName: \"kubernetes.io/projected/6dd379c7-2619-4c19-b4e1-935d438fd8a6-kube-api-access-s7bs6\") pod \"payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq\" (UID: \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:21.122678 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.122571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ppq7\" (UniqueName: \"kubernetes.io/projected/5911ad76-b763-4a9b-8134-206a6da66eb4-kube-api-access-9ppq7\") pod \"payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66\" (UID: \"5911ad76-b763-4a9b-8134-206a6da66eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.123218 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.123200    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd379c7-2619-4c19-b4e1-935d438fd8a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq\" (UID: \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:21.130757 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.130723    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bs6\" (UniqueName: \"kubernetes.io/projected/6dd379c7-2619-4c19-b4e1-935d438fd8a6-kube-api-access-s7bs6\") pod \"payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq\" (UID: \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:21.223539 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.223505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ppq7\" (UniqueName: \"kubernetes.io/projected/5911ad76-b763-4a9b-8134-206a6da66eb4-kube-api-access-9ppq7\") pod \"payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66\" (UID: \"5911ad76-b763-4a9b-8134-206a6da66eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.223696 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.223583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5911ad76-b763-4a9b-8134-206a6da66eb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66\" (UID: \"5911ad76-b763-4a9b-8134-206a6da66eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.224243 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.224225    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5911ad76-b763-4a9b-8134-206a6da66eb4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66\" (UID: \"5911ad76-b763-4a9b-8134-206a6da66eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.232079 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.232062    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ppq7\" (UniqueName: \"kubernetes.io/projected/5911ad76-b763-4a9b-8134-206a6da66eb4-kube-api-access-9ppq7\") pod \"payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66\" (UID: \"5911ad76-b763-4a9b-8134-206a6da66eb4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.304625 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.304563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:21.338448 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.338421    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.577338 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.577249    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"]
Sep 07 21:28:21.579610 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:28:21.579572    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd379c7_2619_4c19_b4e1_935d438fd8a6.slice/crio-301ce8be6fab2b9f25281a7fda9ef349ec9db0e2a9e5291e29d01b4abcedebb7 WatchSource:0}: Error finding container 301ce8be6fab2b9f25281a7fda9ef349ec9db0e2a9e5291e29d01b4abcedebb7: Status 404 returned error can't find the container with id 301ce8be6fab2b9f25281a7fda9ef349ec9db0e2a9e5291e29d01b4abcedebb7
Sep 07 21:28:21.617522 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.617448    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"]
Sep 07 21:28:21.636051 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:28:21.636024    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5911ad76_b763_4a9b_8134_206a6da66eb4.slice/crio-58d7ea457983e6551d264622806068df6bdcc1465eeefecf3b46933ac509d937 WatchSource:0}: Error finding container 58d7ea457983e6551d264622806068df6bdcc1465eeefecf3b46933ac509d937: Status 404 returned error can't find the container with id 58d7ea457983e6551d264622806068df6bdcc1465eeefecf3b46933ac509d937
Sep 07 21:28:21.922212 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.922173    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq" event={"ID":"6dd379c7-2619-4c19-b4e1-935d438fd8a6","Type":"ContainerStarted","Data":"c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03"}
Sep 07 21:28:21.922416 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.922234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq" event={"ID":"6dd379c7-2619-4c19-b4e1-935d438fd8a6","Type":"ContainerStarted","Data":"301ce8be6fab2b9f25281a7fda9ef349ec9db0e2a9e5291e29d01b4abcedebb7"}
Sep 07 21:28:21.922416 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.922263    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:21.923801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.923777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66" event={"ID":"5911ad76-b763-4a9b-8134-206a6da66eb4","Type":"ContainerStarted","Data":"da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709"}
Sep 07 21:28:21.923902 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.923820    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:21.923902 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.923836    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66" event={"ID":"5911ad76-b763-4a9b-8134-206a6da66eb4","Type":"ContainerStarted","Data":"58d7ea457983e6551d264622806068df6bdcc1465eeefecf3b46933ac509d937"}
Sep 07 21:28:21.938359 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.938312    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq" podStartSLOduration=1.938293565 podStartE2EDuration="1.938293565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:28:21.936694032 +0000 UTC m=+1594.018221377" watchObservedRunningTime="2026-09-07 21:28:21.938293565 +0000 UTC m=+1594.019820910"
Sep 07 21:28:21.953565 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:21.953519    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66" podStartSLOduration=0.953502935 podStartE2EDuration="953.502935ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:28:21.951046006 +0000 UTC m=+1594.032573348" watchObservedRunningTime="2026-09-07 21:28:21.953502935 +0000 UTC m=+1594.035030276"
Sep 07 21:28:25.354920 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:25.354877    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:28:32.931868 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:32.931836    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:28:32.932472 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:32.931910    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:28:49.376908 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:49.376861    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66_5911ad76-b763-4a9b-8134-206a6da66eb4/payload-processing/0.log"
Sep 07 21:28:53.255565 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.255521    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"]
Sep 07 21:28:53.258908 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.258887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.261388 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.261359    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-52269e\"/\"kube-root-ca.crt\""
Sep 07 21:28:53.262222 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.262199    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-52269e\"/\"openshift-service-ca.crt\""
Sep 07 21:28:53.262332 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.262225    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-52269e\"/\"default-dockercfg-npvgw\""
Sep 07 21:28:53.262332 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.262225    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-52269e\"/\"ipp-route-52269e-kserve-self-signed-certs\""
Sep 07 21:28:53.269904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.269879    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"]
Sep 07 21:28:53.397671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.397635    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-model-cache\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.397671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.397674    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-dshm\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.397874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.397709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966349c3-145b-4ac0-84ae-e1df7a3f031b-tls-certs\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.397874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.397794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-home\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.397874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.397822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-tmp-dir\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.397874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.397853    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfppj\" (UniqueName: \"kubernetes.io/projected/966349c3-145b-4ac0-84ae-e1df7a3f031b-kube-api-access-wfppj\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.498916 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.498874    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-home\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.498916 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.498921    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-tmp-dir\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.499073 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.498958    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfppj\" (UniqueName: \"kubernetes.io/projected/966349c3-145b-4ac0-84ae-e1df7a3f031b-kube-api-access-wfppj\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.499073 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.499020    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-model-cache\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.499073 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.499050    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-dshm\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.499212 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.499093    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966349c3-145b-4ac0-84ae-e1df7a3f031b-tls-certs\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.499381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.499359    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-home\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.499381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.499373    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-tmp-dir\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.499492 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.499413    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-model-cache\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.501334 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.501311    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-dshm\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.501492 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.501477    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966349c3-145b-4ac0-84ae-e1df7a3f031b-tls-certs\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.507093 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.507039    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfppj\" (UniqueName: \"kubernetes.io/projected/966349c3-145b-4ac0-84ae-e1df7a3f031b-kube-api-access-wfppj\") pod \"ipp-route-52269e-kserve-8477dc647f-8bmf2\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") " pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.570083 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.570046    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:28:53.841354 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:53.841329    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"]
Sep 07 21:28:53.843161 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:28:53.843131    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod966349c3_145b_4ac0_84ae_e1df7a3f031b.slice/crio-41d42c5e648c237d4f722c210a5faae14a0b00751c5603bc178b7b57e217a585 WatchSource:0}: Error finding container 41d42c5e648c237d4f722c210a5faae14a0b00751c5603bc178b7b57e217a585: Status 404 returned error can't find the container with id 41d42c5e648c237d4f722c210a5faae14a0b00751c5603bc178b7b57e217a585
Sep 07 21:28:54.087083 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:54.087048    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2" event={"ID":"966349c3-145b-4ac0-84ae-e1df7a3f031b","Type":"ContainerStarted","Data":"63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4"}
Sep 07 21:28:54.087291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:54.087096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2" event={"ID":"966349c3-145b-4ac0-84ae-e1df7a3f031b","Type":"ContainerStarted","Data":"41d42c5e648c237d4f722c210a5faae14a0b00751c5603bc178b7b57e217a585"}
Sep 07 21:28:54.103004 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:54.102885    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2" podStartSLOduration=1.102872986 podStartE2EDuration="1.102872986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:28:54.102700138 +0000 UTC m=+1626.184227482" watchObservedRunningTime="2026-09-07 21:28:54.102872986 +0000 UTC m=+1626.184400334"
Sep 07 21:28:57.157248 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:28:57.157211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:29:03.570732 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:03.570702    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:29:03.571166 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:03.570744    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:29:03.583211 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:03.583190    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:29:04.146989 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:04.146936    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:29:15.755253 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:15.755216    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:29:19.786673 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:19.786640    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f_b2ab5ee1-6a4f-439a-b0cf-d1131612419e/payload-processing/0.log"
Sep 07 21:29:20.026617 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:20.026588    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66_5911ad76-b763-4a9b-8134-206a6da66eb4/payload-processing/0.log"
Sep 07 21:29:25.410986 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.410937    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"]
Sep 07 21:29:25.414382 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.414364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:25.416825 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.416807    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f420a1\""
Sep 07 21:29:25.416995 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.416958    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f420a1-dockercfg-gjq89\""
Sep 07 21:29:25.424191 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.424165    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"]
Sep 07 21:29:25.478344 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.478319    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tc8w\" (UniqueName: \"kubernetes.io/projected/c5387d9b-c923-46fe-a497-6b0cb6ba9974-kube-api-access-6tc8w\") pod \"payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t\" (UID: \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:25.478457 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.478381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5387d9b-c923-46fe-a497-6b0cb6ba9974-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t\" (UID: \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:25.579282 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.579256    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tc8w\" (UniqueName: \"kubernetes.io/projected/c5387d9b-c923-46fe-a497-6b0cb6ba9974-kube-api-access-6tc8w\") pod \"payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t\" (UID: \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:25.579364 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.579320    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5387d9b-c923-46fe-a497-6b0cb6ba9974-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t\" (UID: \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:25.579999 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.579978    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5387d9b-c923-46fe-a497-6b0cb6ba9974-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t\" (UID: \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:25.586922 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.586895    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tc8w\" (UniqueName: \"kubernetes.io/projected/c5387d9b-c923-46fe-a497-6b0cb6ba9974-kube-api-access-6tc8w\") pod \"payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t\" (UID: \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:25.725680 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.725606    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:25.994151 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:25.994061    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"]
Sep 07 21:29:25.997349 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:29:25.997321    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5387d9b_c923_46fe_a497_6b0cb6ba9974.slice/crio-884659ea0032c982bbf3980f84acb8d1845fc658c3beabefd7796c867287eacd WatchSource:0}: Error finding container 884659ea0032c982bbf3980f84acb8d1845fc658c3beabefd7796c867287eacd: Status 404 returned error can't find the container with id 884659ea0032c982bbf3980f84acb8d1845fc658c3beabefd7796c867287eacd
Sep 07 21:29:26.243980 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:26.243931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t" event={"ID":"c5387d9b-c923-46fe-a497-6b0cb6ba9974","Type":"ContainerStarted","Data":"814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f"}
Sep 07 21:29:26.244140 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:26.244002    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:26.244140 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:26.244019    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t" event={"ID":"c5387d9b-c923-46fe-a497-6b0cb6ba9974","Type":"ContainerStarted","Data":"884659ea0032c982bbf3980f84acb8d1845fc658c3beabefd7796c867287eacd"}
Sep 07 21:29:26.263614 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:26.263525    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t" podStartSLOduration=1.263511977 podStartE2EDuration="1.263511977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:29:26.261184623 +0000 UTC m=+1658.342711965" watchObservedRunningTime="2026-09-07 21:29:26.263511977 +0000 UTC m=+1658.345039317"
Sep 07 21:29:33.053203 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:33.053160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:29:37.252146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:37.252112    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:48.746260 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:48.746228    2565 scope.go:117] "RemoveContainer" containerID="154d4985f05ff19cb7be37d3aae27cad021c5e2d3d5c7634a35a02f757c6ea65"
Sep 07 21:29:49.258656 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.258620    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"]
Sep 07 21:29:49.263366 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.263348    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"
Sep 07 21:29:49.265680 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.265654    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:29:49.265787 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.265688    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:29:49.270685 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.270454    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"]
Sep 07 21:29:49.382810 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.382781    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e762fb1f-4adb-4530-b633-834648c6e338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn\" (UID: \"e762fb1f-4adb-4530-b633-834648c6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"
Sep 07 21:29:49.483444 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.483413    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e762fb1f-4adb-4530-b633-834648c6e338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn\" (UID: \"e762fb1f-4adb-4530-b633-834648c6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"
Sep 07 21:29:49.484070 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.484051    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e762fb1f-4adb-4530-b633-834648c6e338-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn\" (UID: \"e762fb1f-4adb-4530-b633-834648c6e338\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"
Sep 07 21:29:49.575092 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.575030    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"
Sep 07 21:29:49.843042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:49.843006    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"]
Sep 07 21:29:49.845301 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:29:49.845271    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode762fb1f_4adb_4530_b633_834648c6e338.slice/crio-538e1666df0b2065a5efac5e2c99c8ea06863196cc128a1e6d5a6527be8a9076 WatchSource:0}: Error finding container 538e1666df0b2065a5efac5e2c99c8ea06863196cc128a1e6d5a6527be8a9076: Status 404 returned error can't find the container with id 538e1666df0b2065a5efac5e2c99c8ea06863196cc128a1e6d5a6527be8a9076
Sep 07 21:29:50.363717 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:50.363684    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn" event={"ID":"e762fb1f-4adb-4530-b633-834648c6e338","Type":"ContainerStarted","Data":"6c3dc1a5e2063855a4972110ce4c0d3ada2334419cf11995b2c53b2c4f8a54d3"}
Sep 07 21:29:50.363876 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:50.363741    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn" event={"ID":"e762fb1f-4adb-4530-b633-834648c6e338","Type":"ContainerStarted","Data":"538e1666df0b2065a5efac5e2c99c8ea06863196cc128a1e6d5a6527be8a9076"}
Sep 07 21:29:50.379833 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:50.379779    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn" podStartSLOduration=1.3797649330000001 podStartE2EDuration="1.379764933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:29:50.379107854 +0000 UTC m=+1682.460635197" watchObservedRunningTime="2026-09-07 21:29:50.379764933 +0000 UTC m=+1682.461292273"
Sep 07 21:29:51.370164 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:51.370128    2565 generic.go:356] "Generic (PLEG): container finished" podID="e762fb1f-4adb-4530-b633-834648c6e338" containerID="6c3dc1a5e2063855a4972110ce4c0d3ada2334419cf11995b2c53b2c4f8a54d3" exitCode=0
Sep 07 21:29:51.370572 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:51.370176    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn" event={"ID":"e762fb1f-4adb-4530-b633-834648c6e338","Type":"ContainerDied","Data":"6c3dc1a5e2063855a4972110ce4c0d3ada2334419cf11995b2c53b2c4f8a54d3"}
Sep 07 21:29:52.500213 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:52.500190    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"
Sep 07 21:29:52.607843 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:52.607809    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e762fb1f-4adb-4530-b633-834648c6e338-maas-api-service-ca\") pod \"e762fb1f-4adb-4530-b633-834648c6e338\" (UID: \"e762fb1f-4adb-4530-b633-834648c6e338\") "
Sep 07 21:29:52.608177 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:52.608155    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e762fb1f-4adb-4530-b633-834648c6e338-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e762fb1f-4adb-4530-b633-834648c6e338" (UID: "e762fb1f-4adb-4530-b633-834648c6e338"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:29:52.709286 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:52.709212    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e762fb1f-4adb-4530-b633-834648c6e338-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:29:53.381623 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:53.381585    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn" event={"ID":"e762fb1f-4adb-4530-b633-834648c6e338","Type":"ContainerDied","Data":"538e1666df0b2065a5efac5e2c99c8ea06863196cc128a1e6d5a6527be8a9076"}
Sep 07 21:29:53.381623 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:53.381617    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"
Sep 07 21:29:53.381891 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:53.381638    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="538e1666df0b2065a5efac5e2c99c8ea06863196cc128a1e6d5a6527be8a9076"
Sep 07 21:29:55.472741 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.472707    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"]
Sep 07 21:29:55.473201 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.472995    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t" podUID="c5387d9b-c923-46fe-a497-6b0cb6ba9974" containerName="payload-pre-processing" containerID="cri-o://814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f" gracePeriod=30
Sep 07 21:29:55.616686 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.616615    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"]
Sep 07 21:29:55.618776 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.618753    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f420a1-r9hcn"]
Sep 07 21:29:55.734661 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.734608    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:55.838785 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.838753    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tc8w\" (UniqueName: \"kubernetes.io/projected/c5387d9b-c923-46fe-a497-6b0cb6ba9974-kube-api-access-6tc8w\") pod \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\" (UID: \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\") "
Sep 07 21:29:55.838938 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.838798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5387d9b-c923-46fe-a497-6b0cb6ba9974-plugins-config-volume\") pod \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\" (UID: \"c5387d9b-c923-46fe-a497-6b0cb6ba9974\") "
Sep 07 21:29:55.839226 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.839191    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5387d9b-c923-46fe-a497-6b0cb6ba9974-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5387d9b-c923-46fe-a497-6b0cb6ba9974" (UID: "c5387d9b-c923-46fe-a497-6b0cb6ba9974"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:29:55.840851 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.840818    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5387d9b-c923-46fe-a497-6b0cb6ba9974-kube-api-access-6tc8w" (OuterVolumeSpecName: "kube-api-access-6tc8w") pod "c5387d9b-c923-46fe-a497-6b0cb6ba9974" (UID: "c5387d9b-c923-46fe-a497-6b0cb6ba9974"). InnerVolumeSpecName "kube-api-access-6tc8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:29:55.940410 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.940379    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tc8w\" (UniqueName: \"kubernetes.io/projected/c5387d9b-c923-46fe-a497-6b0cb6ba9974-kube-api-access-6tc8w\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:29:55.940410 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:55.940405    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5387d9b-c923-46fe-a497-6b0cb6ba9974-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:29:56.363726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.363692    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e762fb1f-4adb-4530-b633-834648c6e338" path="/var/lib/kubelet/pods/e762fb1f-4adb-4530-b633-834648c6e338/volumes"
Sep 07 21:29:56.398112 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.398085    2565 generic.go:356] "Generic (PLEG): container finished" podID="c5387d9b-c923-46fe-a497-6b0cb6ba9974" containerID="814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f" exitCode=0
Sep 07 21:29:56.398251 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.398126    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"
Sep 07 21:29:56.398251 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.398171    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t" event={"ID":"c5387d9b-c923-46fe-a497-6b0cb6ba9974","Type":"ContainerDied","Data":"814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f"}
Sep 07 21:29:56.398251 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.398217    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t" event={"ID":"c5387d9b-c923-46fe-a497-6b0cb6ba9974","Type":"ContainerDied","Data":"884659ea0032c982bbf3980f84acb8d1845fc658c3beabefd7796c867287eacd"}
Sep 07 21:29:56.398251 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.398246    2565 scope.go:117] "RemoveContainer" containerID="814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f"
Sep 07 21:29:56.407472 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.407455    2565 scope.go:117] "RemoveContainer" containerID="814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f"
Sep 07 21:29:56.407742 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:29:56.407722    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f\": container with ID starting with 814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f not found: ID does not exist" containerID="814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f"
Sep 07 21:29:56.407783 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.407750    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f"} err="failed to get container status \"814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f\": rpc error: code = NotFound desc = could not find container \"814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f\": container with ID starting with 814c1f8bf551f96cc34fe178f8b33901a2bd41cba1905a8a73f3bae548df9c9f not found: ID does not exist"
Sep 07 21:29:56.415120 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.415091    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"]
Sep 07 21:29:56.418663 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:56.418642    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f420a1-c7f7b4bf8-x6t4t"]
Sep 07 21:29:58.369591 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:58.369555    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5387d9b-c923-46fe-a497-6b0cb6ba9974" path="/var/lib/kubelet/pods/c5387d9b-c923-46fe-a497-6b0cb6ba9974/volumes"
Sep 07 21:29:59.648156 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:29:59.648126    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:30:00.136539 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.136500    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"]
Sep 07 21:30:00.136930 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.136893    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e762fb1f-4adb-4530-b633-834648c6e338" containerName="revoke-keys"
Sep 07 21:30:00.137006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.136934    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e762fb1f-4adb-4530-b633-834648c6e338" containerName="revoke-keys"
Sep 07 21:30:00.137006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.136997    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5387d9b-c923-46fe-a497-6b0cb6ba9974" containerName="payload-pre-processing"
Sep 07 21:30:00.137006 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.137005    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5387d9b-c923-46fe-a497-6b0cb6ba9974" containerName="payload-pre-processing"
Sep 07 21:30:00.137118 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.137072    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5387d9b-c923-46fe-a497-6b0cb6ba9974" containerName="payload-pre-processing"
Sep 07 21:30:00.137118 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.137082    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e762fb1f-4adb-4530-b633-834648c6e338" containerName="revoke-keys"
Sep 07 21:30:00.141158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.141142    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn"]
Sep 07 21:30:00.141269 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.141250    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"
Sep 07 21:30:00.143741 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.143725    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:30:00.144103 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.144083    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"]
Sep 07 21:30:00.144214 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.144200    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn"
Sep 07 21:30:00.147143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.147123    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"
Sep 07 21:30:00.148799 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.148773    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"]
Sep 07 21:30:00.151090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.150958    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"]
Sep 07 21:30:00.153227 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.153201    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn"]
Sep 07 21:30:00.155319 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.155299    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"
Sep 07 21:30:00.165219 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.165191    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn"
Sep 07 21:30:00.206635 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.206607    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"
Sep 07 21:30:00.435390 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.435312    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"]
Sep 07 21:30:00.437930 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:30:00.437905    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48085dd2_39d6_44ae_b24c_39061714b799.slice/crio-199fe50c4f74a5148bd1c8a44330ea1a28f1e3ff4792de0133e1b5103b1ac2e8 WatchSource:0}: Error finding container 199fe50c4f74a5148bd1c8a44330ea1a28f1e3ff4792de0133e1b5103b1ac2e8: Status 404 returned error can't find the container with id 199fe50c4f74a5148bd1c8a44330ea1a28f1e3ff4792de0133e1b5103b1ac2e8
Sep 07 21:30:00.452445 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.452423    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn"]
Sep 07 21:30:00.455214 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:30:00.455186    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07490ce7_59b8_404d_9d5d_f4a50083aa01.slice/crio-36fa842d5bb3b35952a86d163e84d5413f3beeac6172f4402e943b6c483fadf0 WatchSource:0}: Error finding container 36fa842d5bb3b35952a86d163e84d5413f3beeac6172f4402e943b6c483fadf0: Status 404 returned error can't find the container with id 36fa842d5bb3b35952a86d163e84d5413f3beeac6172f4402e943b6c483fadf0
Sep 07 21:30:00.503326 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:00.503303    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"]
Sep 07 21:30:00.505163 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:30:00.505139    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93947ab6_fa0d_461c_a96b_dbb4ad6faae8.slice/crio-f22a0d6097b83a043600d46926e8962496610675a46a65c65382fc26989dc592 WatchSource:0}: Error finding container f22a0d6097b83a043600d46926e8962496610675a46a65c65382fc26989dc592: Status 404 returned error can't find the container with id f22a0d6097b83a043600d46926e8962496610675a46a65c65382fc26989dc592
Sep 07 21:30:01.424430 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.424394    2565 generic.go:356] "Generic (PLEG): container finished" podID="93947ab6-fa0d-461c-a96b-dbb4ad6faae8" containerID="5df45a13b7dbd72e0e65ce46fd435c63c6cc22d53f2f4cf7f4a73b863f19219d" exitCode=0
Sep 07 21:30:01.424830 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.424442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564" event={"ID":"93947ab6-fa0d-461c-a96b-dbb4ad6faae8","Type":"ContainerDied","Data":"5df45a13b7dbd72e0e65ce46fd435c63c6cc22d53f2f4cf7f4a73b863f19219d"}
Sep 07 21:30:01.424830 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.424494    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564" event={"ID":"93947ab6-fa0d-461c-a96b-dbb4ad6faae8","Type":"ContainerStarted","Data":"f22a0d6097b83a043600d46926e8962496610675a46a65c65382fc26989dc592"}
Sep 07 21:30:01.426017 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.425988    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb" event={"ID":"48085dd2-39d6-44ae-b24c-39061714b799","Type":"ContainerStarted","Data":"c9a0cd6fd870ae000912a66cd30b8c46036725c2f558b1baf3ca4b66b7f5b9fe"}
Sep 07 21:30:01.426198 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.426033    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb" event={"ID":"48085dd2-39d6-44ae-b24c-39061714b799","Type":"ContainerStarted","Data":"199fe50c4f74a5148bd1c8a44330ea1a28f1e3ff4792de0133e1b5103b1ac2e8"}
Sep 07 21:30:01.427545 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.427519    2565 generic.go:356] "Generic (PLEG): container finished" podID="07490ce7-59b8-404d-9d5d-f4a50083aa01" containerID="5565748cc788b3c8317a7c3bbe141a7297fafac8f0c05a90d3149b86bc84e784" exitCode=0
Sep 07 21:30:01.427658 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.427551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn" event={"ID":"07490ce7-59b8-404d-9d5d-f4a50083aa01","Type":"ContainerDied","Data":"5565748cc788b3c8317a7c3bbe141a7297fafac8f0c05a90d3149b86bc84e784"}
Sep 07 21:30:01.427658 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.427579    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn" event={"ID":"07490ce7-59b8-404d-9d5d-f4a50083aa01","Type":"ContainerStarted","Data":"36fa842d5bb3b35952a86d163e84d5413f3beeac6172f4402e943b6c483fadf0"}
Sep 07 21:30:01.451720 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:01.451675    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb" podStartSLOduration=1.451662456 podStartE2EDuration="1.451662456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:30:01.449760143 +0000 UTC m=+1693.531287485" watchObservedRunningTime="2026-09-07 21:30:01.451662456 +0000 UTC m=+1693.533189796"
Sep 07 21:30:02.437158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:02.437122    2565 generic.go:356] "Generic (PLEG): container finished" podID="48085dd2-39d6-44ae-b24c-39061714b799" containerID="c9a0cd6fd870ae000912a66cd30b8c46036725c2f558b1baf3ca4b66b7f5b9fe" exitCode=0
Sep 07 21:30:02.437636 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:02.437207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb" event={"ID":"48085dd2-39d6-44ae-b24c-39061714b799","Type":"ContainerDied","Data":"c9a0cd6fd870ae000912a66cd30b8c46036725c2f558b1baf3ca4b66b7f5b9fe"}
Sep 07 21:30:02.572214 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:02.572182    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn"
Sep 07 21:30:02.608314 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:02.608292    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"
Sep 07 21:30:03.444381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:03.444350    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn"
Sep 07 21:30:03.444381 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:03.444361    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813610-jrhdn" event={"ID":"07490ce7-59b8-404d-9d5d-f4a50083aa01","Type":"ContainerDied","Data":"36fa842d5bb3b35952a86d163e84d5413f3beeac6172f4402e943b6c483fadf0"}
Sep 07 21:30:03.444913 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:03.444405    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36fa842d5bb3b35952a86d163e84d5413f3beeac6172f4402e943b6c483fadf0"
Sep 07 21:30:03.445710 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:03.445688    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"
Sep 07 21:30:03.445846 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:03.445736    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564" event={"ID":"93947ab6-fa0d-461c-a96b-dbb4ad6faae8","Type":"ContainerDied","Data":"f22a0d6097b83a043600d46926e8962496610675a46a65c65382fc26989dc592"}
Sep 07 21:30:03.445846 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:03.445771    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f22a0d6097b83a043600d46926e8962496610675a46a65c65382fc26989dc592"
Sep 07 21:30:03.575924 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:03.575897    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"
Sep 07 21:30:04.452064 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:04.452029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb" event={"ID":"48085dd2-39d6-44ae-b24c-39061714b799","Type":"ContainerDied","Data":"199fe50c4f74a5148bd1c8a44330ea1a28f1e3ff4792de0133e1b5103b1ac2e8"}
Sep 07 21:30:04.452064 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:04.452060    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"
Sep 07 21:30:04.452469 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:04.452083    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="199fe50c4f74a5148bd1c8a44330ea1a28f1e3ff4792de0133e1b5103b1ac2e8"
Sep 07 21:30:05.700955 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.700923    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"]
Sep 07 21:30:05.701479 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701459    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93947ab6-fa0d-461c-a96b-dbb4ad6faae8" containerName="cleanup"
Sep 07 21:30:05.701563 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701482    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="93947ab6-fa0d-461c-a96b-dbb4ad6faae8" containerName="cleanup"
Sep 07 21:30:05.701563 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701510    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07490ce7-59b8-404d-9d5d-f4a50083aa01" containerName="cleanup"
Sep 07 21:30:05.701563 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701520    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="07490ce7-59b8-404d-9d5d-f4a50083aa01" containerName="cleanup"
Sep 07 21:30:05.701563 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701539    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48085dd2-39d6-44ae-b24c-39061714b799" containerName="cleanup"
Sep 07 21:30:05.701563 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701547    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48085dd2-39d6-44ae-b24c-39061714b799" containerName="cleanup"
Sep 07 21:30:05.701810 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701640    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="93947ab6-fa0d-461c-a96b-dbb4ad6faae8" containerName="cleanup"
Sep 07 21:30:05.701810 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701655    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="07490ce7-59b8-404d-9d5d-f4a50083aa01" containerName="cleanup"
Sep 07 21:30:05.701810 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.701665    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="48085dd2-39d6-44ae-b24c-39061714b799" containerName="cleanup"
Sep 07 21:30:05.705003 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.704981    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"
Sep 07 21:30:05.707309 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.707289    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:30:05.707415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.707327    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:30:05.711567 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.711545    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"]
Sep 07 21:30:05.713197 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.713161    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160280b5-03b0-42dc-a320-1f12e95e7477-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c\" (UID: \"160280b5-03b0-42dc-a320-1f12e95e7477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"
Sep 07 21:30:05.813588 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.813559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160280b5-03b0-42dc-a320-1f12e95e7477-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c\" (UID: \"160280b5-03b0-42dc-a320-1f12e95e7477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"
Sep 07 21:30:05.814801 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:05.814772    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160280b5-03b0-42dc-a320-1f12e95e7477-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c\" (UID: \"160280b5-03b0-42dc-a320-1f12e95e7477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"
Sep 07 21:30:06.017486 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:06.017370    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"
Sep 07 21:30:06.285700 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:06.285627    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"]
Sep 07 21:30:06.295590 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:30:06.295562    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160280b5_03b0_42dc_a320_1f12e95e7477.slice/crio-5112930ae6722908e9fefe39d22a83ae9d7214fb1e9b67d2880325869ac31092 WatchSource:0}: Error finding container 5112930ae6722908e9fefe39d22a83ae9d7214fb1e9b67d2880325869ac31092: Status 404 returned error can't find the container with id 5112930ae6722908e9fefe39d22a83ae9d7214fb1e9b67d2880325869ac31092
Sep 07 21:30:06.464524 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:06.464492    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c" event={"ID":"160280b5-03b0-42dc-a320-1f12e95e7477","Type":"ContainerStarted","Data":"5112930ae6722908e9fefe39d22a83ae9d7214fb1e9b67d2880325869ac31092"}
Sep 07 21:30:07.471024 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:07.470987    2565 generic.go:356] "Generic (PLEG): container finished" podID="160280b5-03b0-42dc-a320-1f12e95e7477" containerID="2094f5b613c99e2d58849d3dac8e7d72c782c14455833796f87e6d031eb97b26" exitCode=0
Sep 07 21:30:07.471427 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:07.471061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c" event={"ID":"160280b5-03b0-42dc-a320-1f12e95e7477","Type":"ContainerDied","Data":"2094f5b613c99e2d58849d3dac8e7d72c782c14455833796f87e6d031eb97b26"}
Sep 07 21:30:08.601279 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:08.601259    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"
Sep 07 21:30:08.633518 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:08.633491    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160280b5-03b0-42dc-a320-1f12e95e7477-maas-api-service-ca\") pod \"160280b5-03b0-42dc-a320-1f12e95e7477\" (UID: \"160280b5-03b0-42dc-a320-1f12e95e7477\") "
Sep 07 21:30:08.633805 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:08.633781    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/160280b5-03b0-42dc-a320-1f12e95e7477-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "160280b5-03b0-42dc-a320-1f12e95e7477" (UID: "160280b5-03b0-42dc-a320-1f12e95e7477"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:30:08.734751 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:08.734677    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/160280b5-03b0-42dc-a320-1f12e95e7477-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:09.482744 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:09.482715    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"
Sep 07 21:30:09.482925 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:09.482714    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c" event={"ID":"160280b5-03b0-42dc-a320-1f12e95e7477","Type":"ContainerDied","Data":"5112930ae6722908e9fefe39d22a83ae9d7214fb1e9b67d2880325869ac31092"}
Sep 07 21:30:09.482925 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:09.482843    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5112930ae6722908e9fefe39d22a83ae9d7214fb1e9b67d2880325869ac31092"
Sep 07 21:30:21.056874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.056790    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:30:21.173117 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.173084    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"]
Sep 07 21:30:21.173350 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.173321    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f" podUID="b2ab5ee1-6a4f-439a-b0cf-d1131612419e" containerName="payload-processing" containerID="cri-o://ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f" gracePeriod=30
Sep 07 21:30:21.351348 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.351316    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"]
Sep 07 21:30:21.354668 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.354643    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-52269e-wq28c"]
Sep 07 21:30:21.421363 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.421337    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:30:21.535657 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.535627    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgtht\" (UniqueName: \"kubernetes.io/projected/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-kube-api-access-wgtht\") pod \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\" (UID: \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\") "
Sep 07 21:30:21.535854 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.535705    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-plugins-config-volume\") pod \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\" (UID: \"b2ab5ee1-6a4f-439a-b0cf-d1131612419e\") "
Sep 07 21:30:21.536118 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.536093    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b2ab5ee1-6a4f-439a-b0cf-d1131612419e" (UID: "b2ab5ee1-6a4f-439a-b0cf-d1131612419e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:30:21.537736 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.537708    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-kube-api-access-wgtht" (OuterVolumeSpecName: "kube-api-access-wgtht") pod "b2ab5ee1-6a4f-439a-b0cf-d1131612419e" (UID: "b2ab5ee1-6a4f-439a-b0cf-d1131612419e"). InnerVolumeSpecName "kube-api-access-wgtht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:30:21.542736 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.542710    2565 generic.go:356] "Generic (PLEG): container finished" podID="b2ab5ee1-6a4f-439a-b0cf-d1131612419e" containerID="ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f" exitCode=0
Sep 07 21:30:21.542870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.542756    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"
Sep 07 21:30:21.542870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.542767    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f" event={"ID":"b2ab5ee1-6a4f-439a-b0cf-d1131612419e","Type":"ContainerDied","Data":"ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f"}
Sep 07 21:30:21.542870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.542838    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f" event={"ID":"b2ab5ee1-6a4f-439a-b0cf-d1131612419e","Type":"ContainerDied","Data":"e05b5aec050d7f8ed71c940924ba6faeed3e46885a715c72f3b4662f26a215ae"}
Sep 07 21:30:21.542870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.542868    2565 scope.go:117] "RemoveContainer" containerID="ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f"
Sep 07 21:30:21.552417 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.552395    2565 scope.go:117] "RemoveContainer" containerID="ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f"
Sep 07 21:30:21.552679 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:30:21.552660    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f\": container with ID starting with ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f not found: ID does not exist" containerID="ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f"
Sep 07 21:30:21.552751 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.552687    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f"} err="failed to get container status \"ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f\": rpc error: code = NotFound desc = could not find container \"ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f\": container with ID starting with ed1d567b34332abae31bac97e66c62f965627912c57113ec21126d3decded18f not found: ID does not exist"
Sep 07 21:30:21.563327 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.563207    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"]
Sep 07 21:30:21.565168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.565145    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-52269e-59c47cc88c-lr24f"]
Sep 07 21:30:21.637028 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.636944    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:21.637028 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:21.636984    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgtht\" (UniqueName: \"kubernetes.io/projected/b2ab5ee1-6a4f-439a-b0cf-d1131612419e-kube-api-access-wgtht\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:22.368868 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:22.368828    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="160280b5-03b0-42dc-a320-1f12e95e7477" path="/var/lib/kubelet/pods/160280b5-03b0-42dc-a320-1f12e95e7477/volumes"
Sep 07 21:30:22.369279 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:22.369221    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ab5ee1-6a4f-439a-b0cf-d1131612419e" path="/var/lib/kubelet/pods/b2ab5ee1-6a4f-439a-b0cf-d1131612419e/volumes"
Sep 07 21:30:26.692845 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:26.692806    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"]
Sep 07 21:30:30.954205 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:30.954160    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:30:41.712843 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:41.712798    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2" podUID="966349c3-145b-4ac0-84ae-e1df7a3f031b" containerName="main" containerID="cri-o://63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4" gracePeriod=45
Sep 07 21:30:41.964051 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:41.963994    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:30:42.111025 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.110991    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-tmp-dir\") pod \"966349c3-145b-4ac0-84ae-e1df7a3f031b\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") "
Sep 07 21:30:42.111194 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111069    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-home\") pod \"966349c3-145b-4ac0-84ae-e1df7a3f031b\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") "
Sep 07 21:30:42.111194 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111087    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfppj\" (UniqueName: \"kubernetes.io/projected/966349c3-145b-4ac0-84ae-e1df7a3f031b-kube-api-access-wfppj\") pod \"966349c3-145b-4ac0-84ae-e1df7a3f031b\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") "
Sep 07 21:30:42.111194 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111133    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-model-cache\") pod \"966349c3-145b-4ac0-84ae-e1df7a3f031b\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") "
Sep 07 21:30:42.111194 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111162    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-dshm\") pod \"966349c3-145b-4ac0-84ae-e1df7a3f031b\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") "
Sep 07 21:30:42.111383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111197    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966349c3-145b-4ac0-84ae-e1df7a3f031b-tls-certs\") pod \"966349c3-145b-4ac0-84ae-e1df7a3f031b\" (UID: \"966349c3-145b-4ac0-84ae-e1df7a3f031b\") "
Sep 07 21:30:42.111383 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111249    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "966349c3-145b-4ac0-84ae-e1df7a3f031b" (UID: "966349c3-145b-4ac0-84ae-e1df7a3f031b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:30:42.111495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111394    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-home" (OuterVolumeSpecName: "home") pod "966349c3-145b-4ac0-84ae-e1df7a3f031b" (UID: "966349c3-145b-4ac0-84ae-e1df7a3f031b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:30:42.111495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111407    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-model-cache" (OuterVolumeSpecName: "model-cache") pod "966349c3-145b-4ac0-84ae-e1df7a3f031b" (UID: "966349c3-145b-4ac0-84ae-e1df7a3f031b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:30:42.111495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111481    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:42.111662 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111498    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:42.111662 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.111507    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:42.113273 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.113240    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-dshm" (OuterVolumeSpecName: "dshm") pod "966349c3-145b-4ac0-84ae-e1df7a3f031b" (UID: "966349c3-145b-4ac0-84ae-e1df7a3f031b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:30:42.113395 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.113367    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966349c3-145b-4ac0-84ae-e1df7a3f031b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "966349c3-145b-4ac0-84ae-e1df7a3f031b" (UID: "966349c3-145b-4ac0-84ae-e1df7a3f031b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:30:42.113454 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.113395    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966349c3-145b-4ac0-84ae-e1df7a3f031b-kube-api-access-wfppj" (OuterVolumeSpecName: "kube-api-access-wfppj") pod "966349c3-145b-4ac0-84ae-e1df7a3f031b" (UID: "966349c3-145b-4ac0-84ae-e1df7a3f031b"). InnerVolumeSpecName "kube-api-access-wfppj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:30:42.212454 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.212422    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/966349c3-145b-4ac0-84ae-e1df7a3f031b-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:42.212454 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.212454    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/966349c3-145b-4ac0-84ae-e1df7a3f031b-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:42.212664 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.212469    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfppj\" (UniqueName: \"kubernetes.io/projected/966349c3-145b-4ac0-84ae-e1df7a3f031b-kube-api-access-wfppj\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:30:42.646127 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.646035    2565 generic.go:356] "Generic (PLEG): container finished" podID="966349c3-145b-4ac0-84ae-e1df7a3f031b" containerID="63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4" exitCode=0
Sep 07 21:30:42.646127 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.646093    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"
Sep 07 21:30:42.646325 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.646124    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2" event={"ID":"966349c3-145b-4ac0-84ae-e1df7a3f031b","Type":"ContainerDied","Data":"63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4"}
Sep 07 21:30:42.646325 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.646170    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2" event={"ID":"966349c3-145b-4ac0-84ae-e1df7a3f031b","Type":"ContainerDied","Data":"41d42c5e648c237d4f722c210a5faae14a0b00751c5603bc178b7b57e217a585"}
Sep 07 21:30:42.646325 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.646190    2565 scope.go:117] "RemoveContainer" containerID="63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4"
Sep 07 21:30:42.654741 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.654721    2565 scope.go:117] "RemoveContainer" containerID="63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4"
Sep 07 21:30:42.655028 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:30:42.655010    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4\": container with ID starting with 63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4 not found: ID does not exist" containerID="63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4"
Sep 07 21:30:42.655076 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.655037    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4"} err="failed to get container status \"63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4\": rpc error: code = NotFound desc = could not find container \"63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4\": container with ID starting with 63156204527aeb8744254f8ec875ab5139bdcc9429d9025ce07d20cabc6605d4 not found: ID does not exist"
Sep 07 21:30:42.661506 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.661484    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"]
Sep 07 21:30:42.665522 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:42.665500    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-52269e/ipp-route-52269e-kserve-8477dc647f-8bmf2"]
Sep 07 21:30:44.364197 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:44.364167    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966349c3-145b-4ac0-84ae-e1df7a3f031b" path="/var/lib/kubelet/pods/966349c3-145b-4ac0-84ae-e1df7a3f031b/volumes"
Sep 07 21:30:48.764721 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:30:48.764691    2565 scope.go:117] "RemoveContainer" containerID="826b9f20165852d4c3f863de3be41009f95372b59b7062706dacfd4cc9ac94f7"
Sep 07 21:31:09.923849 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.923813    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"]
Sep 07 21:31:09.924250 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924165    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160280b5-03b0-42dc-a320-1f12e95e7477" containerName="revoke-keys"
Sep 07 21:31:09.924250 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924177    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="160280b5-03b0-42dc-a320-1f12e95e7477" containerName="revoke-keys"
Sep 07 21:31:09.924250 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924193    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966349c3-145b-4ac0-84ae-e1df7a3f031b" containerName="main"
Sep 07 21:31:09.924250 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924198    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="966349c3-145b-4ac0-84ae-e1df7a3f031b" containerName="main"
Sep 07 21:31:09.924250 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924207    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ab5ee1-6a4f-439a-b0cf-d1131612419e" containerName="payload-processing"
Sep 07 21:31:09.924250 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924212    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ab5ee1-6a4f-439a-b0cf-d1131612419e" containerName="payload-processing"
Sep 07 21:31:09.924447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924268    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="160280b5-03b0-42dc-a320-1f12e95e7477" containerName="revoke-keys"
Sep 07 21:31:09.924447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924277    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2ab5ee1-6a4f-439a-b0cf-d1131612419e" containerName="payload-processing"
Sep 07 21:31:09.924447 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.924285    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="966349c3-145b-4ac0-84ae-e1df7a3f031b" containerName="main"
Sep 07 21:31:09.927271 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.927254    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"
Sep 07 21:31:09.929367 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.929346    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:31:09.929443 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.929406    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:31:09.933886 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.933861    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"]
Sep 07 21:31:09.936778 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:09.936749    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/637e69c1-c564-4f7d-955e-1fe8b804e318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c\" (UID: \"637e69c1-c564-4f7d-955e-1fe8b804e318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"
Sep 07 21:31:10.037824 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:10.037787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/637e69c1-c564-4f7d-955e-1fe8b804e318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c\" (UID: \"637e69c1-c564-4f7d-955e-1fe8b804e318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"
Sep 07 21:31:10.038428 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:10.038408    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/637e69c1-c564-4f7d-955e-1fe8b804e318-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c\" (UID: \"637e69c1-c564-4f7d-955e-1fe8b804e318\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"
Sep 07 21:31:10.238769 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:10.238682    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"
Sep 07 21:31:10.512865 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:10.512839    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"]
Sep 07 21:31:10.517165 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:31:10.517135    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637e69c1_c564_4f7d_955e_1fe8b804e318.slice/crio-6f3aac5158d14754fe297790bd58b61aea31fb02e702eed2abc101c0e341f14f WatchSource:0}: Error finding container 6f3aac5158d14754fe297790bd58b61aea31fb02e702eed2abc101c0e341f14f: Status 404 returned error can't find the container with id 6f3aac5158d14754fe297790bd58b61aea31fb02e702eed2abc101c0e341f14f
Sep 07 21:31:10.785216 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:10.785189    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c" event={"ID":"637e69c1-c564-4f7d-955e-1fe8b804e318","Type":"ContainerStarted","Data":"5c9f50004a15ad8bf3b304f392b4e7e533fd9ede699e073b2c58edd5d8b9f51b"}
Sep 07 21:31:10.785322 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:10.785234    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c" event={"ID":"637e69c1-c564-4f7d-955e-1fe8b804e318","Type":"ContainerStarted","Data":"6f3aac5158d14754fe297790bd58b61aea31fb02e702eed2abc101c0e341f14f"}
Sep 07 21:31:10.799854 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:10.799801    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c" podStartSLOduration=1.799783216 podStartE2EDuration="1.799783216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:31:10.799035716 +0000 UTC m=+1762.880563053" watchObservedRunningTime="2026-09-07 21:31:10.799783216 +0000 UTC m=+1762.881310557"
Sep 07 21:31:11.791713 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:11.791673    2565 generic.go:356] "Generic (PLEG): container finished" podID="637e69c1-c564-4f7d-955e-1fe8b804e318" containerID="5c9f50004a15ad8bf3b304f392b4e7e533fd9ede699e073b2c58edd5d8b9f51b" exitCode=0
Sep 07 21:31:11.792146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:11.791782    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c" event={"ID":"637e69c1-c564-4f7d-955e-1fe8b804e318","Type":"ContainerDied","Data":"5c9f50004a15ad8bf3b304f392b4e7e533fd9ede699e073b2c58edd5d8b9f51b"}
Sep 07 21:31:12.924139 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:12.924112    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"
Sep 07 21:31:12.959831 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:12.959801    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/637e69c1-c564-4f7d-955e-1fe8b804e318-maas-api-service-ca\") pod \"637e69c1-c564-4f7d-955e-1fe8b804e318\" (UID: \"637e69c1-c564-4f7d-955e-1fe8b804e318\") "
Sep 07 21:31:12.960214 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:12.960186    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637e69c1-c564-4f7d-955e-1fe8b804e318-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "637e69c1-c564-4f7d-955e-1fe8b804e318" (UID: "637e69c1-c564-4f7d-955e-1fe8b804e318"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:31:13.060671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:13.060585    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/637e69c1-c564-4f7d-955e-1fe8b804e318-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:13.802499 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:13.802466    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"
Sep 07 21:31:13.802682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:13.802496    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c" event={"ID":"637e69c1-c564-4f7d-955e-1fe8b804e318","Type":"ContainerDied","Data":"6f3aac5158d14754fe297790bd58b61aea31fb02e702eed2abc101c0e341f14f"}
Sep 07 21:31:13.802682 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:13.802570    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f3aac5158d14754fe297790bd58b61aea31fb02e702eed2abc101c0e341f14f"
Sep 07 21:31:20.411604 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.411567    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"]
Sep 07 21:31:20.412066 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.411808    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66" podUID="5911ad76-b763-4a9b-8134-206a6da66eb4" containerName="payload-processing" containerID="cri-o://da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709" gracePeriod=30
Sep 07 21:31:20.419225 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.419198    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"]
Sep 07 21:31:20.419476 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.419425    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq" podUID="6dd379c7-2619-4c19-b4e1-935d438fd8a6" containerName="payload-pre-processing" containerID="cri-o://c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03" gracePeriod=30
Sep 07 21:31:20.560610 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.560572    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"]
Sep 07 21:31:20.565374 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.565339    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-629e95-68v2c"]
Sep 07 21:31:20.682512 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.682485    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:31:20.685187 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.685167    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:31:20.724581 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.724557    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ppq7\" (UniqueName: \"kubernetes.io/projected/5911ad76-b763-4a9b-8134-206a6da66eb4-kube-api-access-9ppq7\") pod \"5911ad76-b763-4a9b-8134-206a6da66eb4\" (UID: \"5911ad76-b763-4a9b-8134-206a6da66eb4\") "
Sep 07 21:31:20.724748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.724609    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7bs6\" (UniqueName: \"kubernetes.io/projected/6dd379c7-2619-4c19-b4e1-935d438fd8a6-kube-api-access-s7bs6\") pod \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\" (UID: \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\") "
Sep 07 21:31:20.724748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.724649    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5911ad76-b763-4a9b-8134-206a6da66eb4-plugins-config-volume\") pod \"5911ad76-b763-4a9b-8134-206a6da66eb4\" (UID: \"5911ad76-b763-4a9b-8134-206a6da66eb4\") "
Sep 07 21:31:20.724748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.724679    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd379c7-2619-4c19-b4e1-935d438fd8a6-plugins-config-volume\") pod \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\" (UID: \"6dd379c7-2619-4c19-b4e1-935d438fd8a6\") "
Sep 07 21:31:20.725093 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.725058    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5911ad76-b763-4a9b-8134-206a6da66eb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5911ad76-b763-4a9b-8134-206a6da66eb4" (UID: "5911ad76-b763-4a9b-8134-206a6da66eb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:31:20.725202 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.725109    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd379c7-2619-4c19-b4e1-935d438fd8a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dd379c7-2619-4c19-b4e1-935d438fd8a6" (UID: "6dd379c7-2619-4c19-b4e1-935d438fd8a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:31:20.726705 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.726685    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5911ad76-b763-4a9b-8134-206a6da66eb4-kube-api-access-9ppq7" (OuterVolumeSpecName: "kube-api-access-9ppq7") pod "5911ad76-b763-4a9b-8134-206a6da66eb4" (UID: "5911ad76-b763-4a9b-8134-206a6da66eb4"). InnerVolumeSpecName "kube-api-access-9ppq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:31:20.726773 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.726720    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd379c7-2619-4c19-b4e1-935d438fd8a6-kube-api-access-s7bs6" (OuterVolumeSpecName: "kube-api-access-s7bs6") pod "6dd379c7-2619-4c19-b4e1-935d438fd8a6" (UID: "6dd379c7-2619-4c19-b4e1-935d438fd8a6"). InnerVolumeSpecName "kube-api-access-s7bs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:31:20.825755 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.825725    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ppq7\" (UniqueName: \"kubernetes.io/projected/5911ad76-b763-4a9b-8134-206a6da66eb4-kube-api-access-9ppq7\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:20.825755 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.825749    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7bs6\" (UniqueName: \"kubernetes.io/projected/6dd379c7-2619-4c19-b4e1-935d438fd8a6-kube-api-access-s7bs6\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:20.825755 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.825759    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5911ad76-b763-4a9b-8134-206a6da66eb4-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:20.825954 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.825768    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd379c7-2619-4c19-b4e1-935d438fd8a6-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:20.838826 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.838798    2565 generic.go:356] "Generic (PLEG): container finished" podID="6dd379c7-2619-4c19-b4e1-935d438fd8a6" containerID="c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03" exitCode=0
Sep 07 21:31:20.838950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.838844    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"
Sep 07 21:31:20.838950 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.838887    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq" event={"ID":"6dd379c7-2619-4c19-b4e1-935d438fd8a6","Type":"ContainerDied","Data":"c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03"}
Sep 07 21:31:20.839090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.838946    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq" event={"ID":"6dd379c7-2619-4c19-b4e1-935d438fd8a6","Type":"ContainerDied","Data":"301ce8be6fab2b9f25281a7fda9ef349ec9db0e2a9e5291e29d01b4abcedebb7"}
Sep 07 21:31:20.839090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.839004    2565 scope.go:117] "RemoveContainer" containerID="c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03"
Sep 07 21:31:20.840313 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.840283    2565 generic.go:356] "Generic (PLEG): container finished" podID="5911ad76-b763-4a9b-8134-206a6da66eb4" containerID="da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709" exitCode=0
Sep 07 21:31:20.840422 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.840326    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"
Sep 07 21:31:20.840422 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.840338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66" event={"ID":"5911ad76-b763-4a9b-8134-206a6da66eb4","Type":"ContainerDied","Data":"da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709"}
Sep 07 21:31:20.840422 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.840386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66" event={"ID":"5911ad76-b763-4a9b-8134-206a6da66eb4","Type":"ContainerDied","Data":"58d7ea457983e6551d264622806068df6bdcc1465eeefecf3b46933ac509d937"}
Sep 07 21:31:20.848035 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.847920    2565 scope.go:117] "RemoveContainer" containerID="c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03"
Sep 07 21:31:20.848289 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:31:20.848261    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03\": container with ID starting with c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03 not found: ID does not exist" containerID="c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03"
Sep 07 21:31:20.848392 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.848298    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03"} err="failed to get container status \"c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03\": rpc error: code = NotFound desc = could not find container \"c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03\": container with ID starting with c102fc854dc49267d8a560293eac723d0d577da4f2c9663b8127034968fefd03 not found: ID does not exist"
Sep 07 21:31:20.848392 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.848315    2565 scope.go:117] "RemoveContainer" containerID="da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709"
Sep 07 21:31:20.856091 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.856073    2565 scope.go:117] "RemoveContainer" containerID="da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709"
Sep 07 21:31:20.856351 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:31:20.856325    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709\": container with ID starting with da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709 not found: ID does not exist" containerID="da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709"
Sep 07 21:31:20.856400 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.856355    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709"} err="failed to get container status \"da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709\": rpc error: code = NotFound desc = could not find container \"da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709\": container with ID starting with da89a17aa07159466a011c393e4aa043fdddf07788d22584aa14a3e897776709 not found: ID does not exist"
Sep 07 21:31:20.860893 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.860873    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"]
Sep 07 21:31:20.865092 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.865071    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-629e95-878687946-knwsq"]
Sep 07 21:31:20.874662 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.874643    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"]
Sep 07 21:31:20.880331 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:20.880311    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-629e95-b4f9676cf-rqp66"]
Sep 07 21:31:22.364086 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:22.364055    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5911ad76-b763-4a9b-8134-206a6da66eb4" path="/var/lib/kubelet/pods/5911ad76-b763-4a9b-8134-206a6da66eb4/volumes"
Sep 07 21:31:22.364486 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:22.364388    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637e69c1-c564-4f7d-955e-1fe8b804e318" path="/var/lib/kubelet/pods/637e69c1-c564-4f7d-955e-1fe8b804e318/volumes"
Sep 07 21:31:22.364703 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:22.364689    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dd379c7-2619-4c19-b4e1-935d438fd8a6" path="/var/lib/kubelet/pods/6dd379c7-2619-4c19-b4e1-935d438fd8a6/volumes"
Sep 07 21:31:24.753289 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:24.753251    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:31:27.379980 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.379933    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"]
Sep 07 21:31:27.380334 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380259    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5911ad76-b763-4a9b-8134-206a6da66eb4" containerName="payload-processing"
Sep 07 21:31:27.380334 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380270    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5911ad76-b763-4a9b-8134-206a6da66eb4" containerName="payload-processing"
Sep 07 21:31:27.380334 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380285    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd379c7-2619-4c19-b4e1-935d438fd8a6" containerName="payload-pre-processing"
Sep 07 21:31:27.380334 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380290    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd379c7-2619-4c19-b4e1-935d438fd8a6" containerName="payload-pre-processing"
Sep 07 21:31:27.380334 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380301    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637e69c1-c564-4f7d-955e-1fe8b804e318" containerName="revoke-keys"
Sep 07 21:31:27.380334 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380307    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="637e69c1-c564-4f7d-955e-1fe8b804e318" containerName="revoke-keys"
Sep 07 21:31:27.380550 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380356    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="637e69c1-c564-4f7d-955e-1fe8b804e318" containerName="revoke-keys"
Sep 07 21:31:27.380550 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380364    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd379c7-2619-4c19-b4e1-935d438fd8a6" containerName="payload-pre-processing"
Sep 07 21:31:27.380550 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.380370    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5911ad76-b763-4a9b-8134-206a6da66eb4" containerName="payload-processing"
Sep 07 21:31:27.384464 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.384444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"
Sep 07 21:31:27.386809 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.386790    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2mv4w\""
Sep 07 21:31:27.386917 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.386790    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 21:31:27.390048 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.390024    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"]
Sep 07 21:31:27.483958 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.483924    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2e45c2-8da4-4a78-876b-91441b80b554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp\" (UID: \"bc2e45c2-8da4-4a78-876b-91441b80b554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"
Sep 07 21:31:27.584435 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.584405    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2e45c2-8da4-4a78-876b-91441b80b554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp\" (UID: \"bc2e45c2-8da4-4a78-876b-91441b80b554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"
Sep 07 21:31:27.585032 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.585012    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2e45c2-8da4-4a78-876b-91441b80b554-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp\" (UID: \"bc2e45c2-8da4-4a78-876b-91441b80b554\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"
Sep 07 21:31:27.696419 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.696341    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"
Sep 07 21:31:27.963747 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:27.963666    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"]
Sep 07 21:31:27.966931 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:31:27.966904    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2e45c2_8da4_4a78_876b_91441b80b554.slice/crio-2239ca85559e7c8739898bc9383b593e2d76c8d4d32e99227becd20857a0752f WatchSource:0}: Error finding container 2239ca85559e7c8739898bc9383b593e2d76c8d4d32e99227becd20857a0752f: Status 404 returned error can't find the container with id 2239ca85559e7c8739898bc9383b593e2d76c8d4d32e99227becd20857a0752f
Sep 07 21:31:28.891403 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:28.891362    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc2e45c2-8da4-4a78-876b-91441b80b554" containerID="cd7e4875db9443a5ec1b1c3236c0a4c52c092d91f746b16b86ee16bd3eda6c03" exitCode=0
Sep 07 21:31:28.891832 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:28.891450    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp" event={"ID":"bc2e45c2-8da4-4a78-876b-91441b80b554","Type":"ContainerDied","Data":"cd7e4875db9443a5ec1b1c3236c0a4c52c092d91f746b16b86ee16bd3eda6c03"}
Sep 07 21:31:28.891832 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:28.891510    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp" event={"ID":"bc2e45c2-8da4-4a78-876b-91441b80b554","Type":"ContainerStarted","Data":"2239ca85559e7c8739898bc9383b593e2d76c8d4d32e99227becd20857a0752f"}
Sep 07 21:31:30.021090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:30.021068    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"
Sep 07 21:31:30.107538 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:30.107489    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2e45c2-8da4-4a78-876b-91441b80b554-maas-api-service-ca\") pod \"bc2e45c2-8da4-4a78-876b-91441b80b554\" (UID: \"bc2e45c2-8da4-4a78-876b-91441b80b554\") "
Sep 07 21:31:30.109403 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:30.108281    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2e45c2-8da4-4a78-876b-91441b80b554-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bc2e45c2-8da4-4a78-876b-91441b80b554" (UID: "bc2e45c2-8da4-4a78-876b-91441b80b554"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:31:30.208493 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:30.208405    2565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc2e45c2-8da4-4a78-876b-91441b80b554-maas-api-service-ca\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:30.902416 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:30.902384    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"
Sep 07 21:31:30.902605 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:30.902417    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp" event={"ID":"bc2e45c2-8da4-4a78-876b-91441b80b554","Type":"ContainerDied","Data":"2239ca85559e7c8739898bc9383b593e2d76c8d4d32e99227becd20857a0752f"}
Sep 07 21:31:30.902605 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:30.902469    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2239ca85559e7c8739898bc9383b593e2d76c8d4d32e99227becd20857a0752f"
Sep 07 21:31:37.943981 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:37.943927    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"]
Sep 07 21:31:37.950492 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:37.950459    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-15d5e2-29813610-nsfmb"]
Sep 07 21:31:37.972091 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:37.972062    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"]
Sep 07 21:31:37.972322 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:37.972297    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5" podUID="68f9e1b0-fb24-478d-b574-5f0669fa4a4f" containerName="payload-processing" containerID="cri-o://5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035" gracePeriod=30
Sep 07 21:31:38.125575 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.125532    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"]
Sep 07 21:31:38.130020 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.129989    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-15d5e2-6qmcp"]
Sep 07 21:31:38.224135 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.224109    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:31:38.364936 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.364905    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48085dd2-39d6-44ae-b24c-39061714b799" path="/var/lib/kubelet/pods/48085dd2-39d6-44ae-b24c-39061714b799/volumes"
Sep 07 21:31:38.365246 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.365232    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc2e45c2-8da4-4a78-876b-91441b80b554" path="/var/lib/kubelet/pods/bc2e45c2-8da4-4a78-876b-91441b80b554/volumes"
Sep 07 21:31:38.379290 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.379268    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r224h\" (UniqueName: \"kubernetes.io/projected/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-kube-api-access-r224h\") pod \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\" (UID: \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\") "
Sep 07 21:31:38.379401 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.379298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-plugins-config-volume\") pod \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\" (UID: \"68f9e1b0-fb24-478d-b574-5f0669fa4a4f\") "
Sep 07 21:31:38.379679 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.379662    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68f9e1b0-fb24-478d-b574-5f0669fa4a4f" (UID: "68f9e1b0-fb24-478d-b574-5f0669fa4a4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:31:38.381472 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.381447    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-kube-api-access-r224h" (OuterVolumeSpecName: "kube-api-access-r224h") pod "68f9e1b0-fb24-478d-b574-5f0669fa4a4f" (UID: "68f9e1b0-fb24-478d-b574-5f0669fa4a4f"). InnerVolumeSpecName "kube-api-access-r224h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:31:38.480895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.480817    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r224h\" (UniqueName: \"kubernetes.io/projected/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-kube-api-access-r224h\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:38.480895 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.480844    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9e1b0-fb24-478d-b574-5f0669fa4a4f-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:38.944161 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.944126    2565 generic.go:356] "Generic (PLEG): container finished" podID="68f9e1b0-fb24-478d-b574-5f0669fa4a4f" containerID="5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035" exitCode=0
Sep 07 21:31:38.944581 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.944172    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"
Sep 07 21:31:38.944581 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.944221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5" event={"ID":"68f9e1b0-fb24-478d-b574-5f0669fa4a4f","Type":"ContainerDied","Data":"5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035"}
Sep 07 21:31:38.944581 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.944273    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5" event={"ID":"68f9e1b0-fb24-478d-b574-5f0669fa4a4f","Type":"ContainerDied","Data":"f9ae212237284b0ef083537c79c3dae08444e6ca4a2b6175e40fdc35724e1584"}
Sep 07 21:31:38.944581 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.944296    2565 scope.go:117] "RemoveContainer" containerID="5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035"
Sep 07 21:31:38.953618 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.953600    2565 scope.go:117] "RemoveContainer" containerID="5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035"
Sep 07 21:31:38.953884 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:31:38.953863    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035\": container with ID starting with 5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035 not found: ID does not exist" containerID="5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035"
Sep 07 21:31:38.953947 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.953897    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035"} err="failed to get container status \"5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035\": rpc error: code = NotFound desc = could not find container \"5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035\": container with ID starting with 5e55214c8059aaf6826ed81d1a7082c2d798c78ec9274d8a2267db76fdd2b035 not found: ID does not exist"
Sep 07 21:31:38.965758 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.965736    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"]
Sep 07 21:31:38.967196 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:38.967177    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-15d5e2-6b6855546c-zn4t5"]
Sep 07 21:31:40.363114 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:40.363075    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68f9e1b0-fb24-478d-b574-5f0669fa4a4f" path="/var/lib/kubelet/pods/68f9e1b0-fb24-478d-b574-5f0669fa4a4f/volumes"
Sep 07 21:31:41.555792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:41.555759    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:31:55.946352 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:55.946310    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"]
Sep 07 21:31:55.950874 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:55.950842    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-7fa21f-29813610-n6564"]
Sep 07 21:31:55.985440 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:55.985404    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"]
Sep 07 21:31:55.985730 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:55.985678    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p" podUID="0a318069-92cc-46b4-add9-3292559a9172" containerName="payload-pre-processing" containerID="cri-o://9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186" gracePeriod=30
Sep 07 21:31:56.226893 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:56.226869    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:31:56.336805 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:56.336762    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a318069-92cc-46b4-add9-3292559a9172-plugins-config-volume\") pod \"0a318069-92cc-46b4-add9-3292559a9172\" (UID: \"0a318069-92cc-46b4-add9-3292559a9172\") "
Sep 07 21:31:56.337009 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:56.336862    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmbcr\" (UniqueName: \"kubernetes.io/projected/0a318069-92cc-46b4-add9-3292559a9172-kube-api-access-dmbcr\") pod \"0a318069-92cc-46b4-add9-3292559a9172\" (UID: \"0a318069-92cc-46b4-add9-3292559a9172\") "
Sep 07 21:31:56.337155 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:56.337130    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a318069-92cc-46b4-add9-3292559a9172-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a318069-92cc-46b4-add9-3292559a9172" (UID: "0a318069-92cc-46b4-add9-3292559a9172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 21:31:56.338939 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:56.338916    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a318069-92cc-46b4-add9-3292559a9172-kube-api-access-dmbcr" (OuterVolumeSpecName: "kube-api-access-dmbcr") pod "0a318069-92cc-46b4-add9-3292559a9172" (UID: "0a318069-92cc-46b4-add9-3292559a9172"). InnerVolumeSpecName "kube-api-access-dmbcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:31:56.363625 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:56.363597    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93947ab6-fa0d-461c-a96b-dbb4ad6faae8" path="/var/lib/kubelet/pods/93947ab6-fa0d-461c-a96b-dbb4ad6faae8/volumes"
Sep 07 21:31:56.438113 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:56.438077    2565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a318069-92cc-46b4-add9-3292559a9172-plugins-config-volume\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:56.438113 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:56.438108    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmbcr\" (UniqueName: \"kubernetes.io/projected/0a318069-92cc-46b4-add9-3292559a9172-kube-api-access-dmbcr\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:31:57.034740 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.034704    2565 generic.go:356] "Generic (PLEG): container finished" podID="0a318069-92cc-46b4-add9-3292559a9172" containerID="9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186" exitCode=0
Sep 07 21:31:57.035167 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.034749    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"
Sep 07 21:31:57.035167 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.034777    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p" event={"ID":"0a318069-92cc-46b4-add9-3292559a9172","Type":"ContainerDied","Data":"9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186"}
Sep 07 21:31:57.035167 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.034820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p" event={"ID":"0a318069-92cc-46b4-add9-3292559a9172","Type":"ContainerDied","Data":"173f2e6a81826bad86c0ec94eb59c2eb47c2d21b0cd1579cfc7a262eb28d508e"}
Sep 07 21:31:57.035167 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.034840    2565 scope.go:117] "RemoveContainer" containerID="9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186"
Sep 07 21:31:57.043562 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.043544    2565 scope.go:117] "RemoveContainer" containerID="9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186"
Sep 07 21:31:57.043831 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:31:57.043813    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186\": container with ID starting with 9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186 not found: ID does not exist" containerID="9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186"
Sep 07 21:31:57.043906 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.043839    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186"} err="failed to get container status \"9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186\": rpc error: code = NotFound desc = could not find container \"9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186\": container with ID starting with 9daf93b8e0fbe3e37a979b8c701f25233fc20f9c54dc98d17373f39561139186 not found: ID does not exist"
Sep 07 21:31:57.051169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.051143    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"]
Sep 07 21:31:57.056778 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:57.056752    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-7fa21f-5c8cb6579d-z927p"]
Sep 07 21:31:58.365382 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:58.365347    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a318069-92cc-46b4-add9-3292559a9172" path="/var/lib/kubelet/pods/0a318069-92cc-46b4-add9-3292559a9172/volumes"
Sep 07 21:31:58.455274 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:31:58.455243    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:32:06.851879 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:32:06.851837    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:32:48.817483 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:32:48.817440    2565 scope.go:117] "RemoveContainer" containerID="0f0af791326d91f4603877cc95fc2f8d662f9be30dbb4f2407c41517a97cb35f"
Sep 07 21:32:48.825748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:32:48.825729    2565 scope.go:117] "RemoveContainer" containerID="efab0cd7c633ba0086432278da3298404f90a194f4a391f3adb9591c2863cd28"
Sep 07 21:32:48.832911 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:32:48.832893    2565 scope.go:117] "RemoveContainer" containerID="498dadbc525479a3c9d821506874fbc619cae1b761c069690369ac77508b64a3"
Sep 07 21:32:48.840236 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:32:48.840222    2565 scope.go:117] "RemoveContainer" containerID="5f4037ea89413882ecbeab1559817aeab271567d9eb613cc79bc04fe09e626b4"
Sep 07 21:33:28.752741 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:33:28.752694    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:33:48.873904 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:33:48.873852    2565 scope.go:117] "RemoveContainer" containerID="6cafd69d43edc470404ae88b2de74a976dbfeeaa991fffd61a1a729fa8528764"
Sep 07 21:33:48.881881 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:33:48.881860    2565 scope.go:117] "RemoveContainer" containerID="698af8fba4c539de59b69a395bdd20a59693650e289163d14149910e45f6da8f"
Sep 07 21:33:57.857325 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:33:57.857288    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:34:05.967836 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:34:05.967798    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:34:13.074781 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:34:13.074748    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:34:21.974991 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:34:21.974942    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:34:29.171763 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:34:29.171726    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:34:36.865651 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:34:36.865614    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:34:46.720173 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:34:46.720134    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:36:15.256485 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:15.256400    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:36:21.953597 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:21.953559    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:36:29.357444 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:29.357406    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:36:30.048552 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048516    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"]
Sep 07 21:36:30.048862 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048848    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a318069-92cc-46b4-add9-3292559a9172" containerName="payload-pre-processing"
Sep 07 21:36:30.048937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048865    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a318069-92cc-46b4-add9-3292559a9172" containerName="payload-pre-processing"
Sep 07 21:36:30.048937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048876    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2e45c2-8da4-4a78-876b-91441b80b554" containerName="revoke-keys"
Sep 07 21:36:30.048937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048882    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2e45c2-8da4-4a78-876b-91441b80b554" containerName="revoke-keys"
Sep 07 21:36:30.048937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048893    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68f9e1b0-fb24-478d-b574-5f0669fa4a4f" containerName="payload-processing"
Sep 07 21:36:30.048937 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048899    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="68f9e1b0-fb24-478d-b574-5f0669fa4a4f" containerName="payload-processing"
Sep 07 21:36:30.049195 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048951    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2e45c2-8da4-4a78-876b-91441b80b554" containerName="revoke-keys"
Sep 07 21:36:30.049195 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048976    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a318069-92cc-46b4-add9-3292559a9172" containerName="payload-pre-processing"
Sep 07 21:36:30.049195 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.048987    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="68f9e1b0-fb24-478d-b574-5f0669fa4a4f" containerName="payload-processing"
Sep 07 21:36:30.051891 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.051871    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.054404 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.054381    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8258edf4-kserve-self-signed-certs\""
Sep 07 21:36:30.063506 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.063485    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"]
Sep 07 21:36:30.121143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.121110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rncc4\" (UniqueName: \"kubernetes.io/projected/225921b1-32ce-419b-8372-c79426b60103-kube-api-access-rncc4\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.121143 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.121144    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-home\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.121335 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.121180    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-dshm\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.121335 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.121251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225921b1-32ce-419b-8372-c79426b60103-tls-certs\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.121415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.121326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-model-cache\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.121415 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.121363    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-tmp-dir\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222056 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-tmp-dir\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222079    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rncc4\" (UniqueName: \"kubernetes.io/projected/225921b1-32ce-419b-8372-c79426b60103-kube-api-access-rncc4\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-home\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222128    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-dshm\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222147    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225921b1-32ce-419b-8372-c79426b60103-tls-certs\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222190 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222182    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-model-cache\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222533 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222504    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-tmp-dir\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222607 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222514    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-home\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.222607 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.222594    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-model-cache\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.224474 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.224453    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-dshm\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.224715 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.224695    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225921b1-32ce-419b-8372-c79426b60103-tls-certs\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.232738 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.232719    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rncc4\" (UniqueName: \"kubernetes.io/projected/225921b1-32ce-419b-8372-c79426b60103-kube-api-access-rncc4\") pod \"e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") " pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.360892 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.360861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:30.632180 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:30.632141    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"]
Sep 07 21:36:30.634701 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:36:30.634669    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod225921b1_32ce_419b_8372_c79426b60103.slice/crio-123c069d0e47bf296df1e4fe8fe8ed34fe510014764c96b6f1373809309f40ae WatchSource:0}: Error finding container 123c069d0e47bf296df1e4fe8fe8ed34fe510014764c96b6f1373809309f40ae: Status 404 returned error can't find the container with id 123c069d0e47bf296df1e4fe8fe8ed34fe510014764c96b6f1373809309f40ae
Sep 07 21:36:31.329025 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:31.328982    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g" event={"ID":"225921b1-32ce-419b-8372-c79426b60103","Type":"ContainerStarted","Data":"3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f"}
Sep 07 21:36:31.329206 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:31.329032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g" event={"ID":"225921b1-32ce-419b-8372-c79426b60103","Type":"ContainerStarted","Data":"123c069d0e47bf296df1e4fe8fe8ed34fe510014764c96b6f1373809309f40ae"}
Sep 07 21:36:31.351171 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:31.346565    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g" podStartSLOduration=1.346532429 podStartE2EDuration="1.346532429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:36:31.345600597 +0000 UTC m=+2083.427127939" watchObservedRunningTime="2026-09-07 21:36:31.346532429 +0000 UTC m=+2083.428059772"
Sep 07 21:36:35.337179 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.337144    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"]
Sep 07 21:36:35.340400 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.340382    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.342637 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.342616    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8258edf4-kserve-self-signed-certs\""
Sep 07 21:36:35.349237 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.349217    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"]
Sep 07 21:36:35.367620 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.367588    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-model-cache\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.367620 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.367619    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-home\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.367797 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.367641    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-tmp-dir\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.367797 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.367658    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-dshm\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.367797 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.367676    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drg5b\" (UniqueName: \"kubernetes.io/projected/426cb757-74ee-4225-a58b-341a7474f8bf-kube-api-access-drg5b\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.367797 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.367780    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426cb757-74ee-4225-a58b-341a7474f8bf-tls-certs\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468429 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468396    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-model-cache\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468429 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-home\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-tmp-dir\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468603 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468569    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-dshm\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468679 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468605    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drg5b\" (UniqueName: \"kubernetes.io/projected/426cb757-74ee-4225-a58b-341a7474f8bf-kube-api-access-drg5b\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468755 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426cb757-74ee-4225-a58b-341a7474f8bf-tls-certs\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468837 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468810    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-home\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468844    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-model-cache\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.468897 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.468882    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-tmp-dir\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.470667 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.470650    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-dshm\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.471146 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.471124    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426cb757-74ee-4225-a58b-341a7474f8bf-tls-certs\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.476735 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.476716    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drg5b\" (UniqueName: \"kubernetes.io/projected/426cb757-74ee-4225-a58b-341a7474f8bf-kube-api-access-drg5b\") pod \"e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") " pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.651029 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.650937    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:35.921271 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:35.921246    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"]
Sep 07 21:36:35.923530 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:36:35.923496    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod426cb757_74ee_4225_a58b_341a7474f8bf.slice/crio-1e4854871c12d200b2af87db8a6bdbc47fbce5d99136fc0b73efaa36e559f6f6 WatchSource:0}: Error finding container 1e4854871c12d200b2af87db8a6bdbc47fbce5d99136fc0b73efaa36e559f6f6: Status 404 returned error can't find the container with id 1e4854871c12d200b2af87db8a6bdbc47fbce5d99136fc0b73efaa36e559f6f6
Sep 07 21:36:36.355104 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:36.355069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46" event={"ID":"426cb757-74ee-4225-a58b-341a7474f8bf","Type":"ContainerStarted","Data":"91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1"}
Sep 07 21:36:36.355104 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:36.355108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46" event={"ID":"426cb757-74ee-4225-a58b-341a7474f8bf","Type":"ContainerStarted","Data":"1e4854871c12d200b2af87db8a6bdbc47fbce5d99136fc0b73efaa36e559f6f6"}
Sep 07 21:36:36.371772 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:36.371715    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46" podStartSLOduration=1.371695503 podStartE2EDuration="1.371695503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:36:36.370283874 +0000 UTC m=+2088.451811215" watchObservedRunningTime="2026-09-07 21:36:36.371695503 +0000 UTC m=+2088.453222845"
Sep 07 21:36:38.400320 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:38.400280    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:36:40.363760 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:40.363726    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:40.364169 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:40.363770    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:40.374454 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:40.374431    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:40.387168 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:40.387147    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:36:45.651445 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:45.651409    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:45.651445 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:45.651459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:45.663906 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:45.663882    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:46.413890 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:46.413859    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:36:47.971072 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:47.971036    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"]
Sep 07 21:36:48.924887 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:48.924843    2565 scope.go:117] "RemoveContainer" containerID="6c3dc1a5e2063855a4972110ce4c0d3ada2334419cf11995b2c53b2c4f8a54d3"
Sep 07 21:36:48.925154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:48.925128    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"]
Sep 07 21:36:48.935517 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:48.935497    2565 scope.go:117] "RemoveContainer" containerID="2094f5b613c99e2d58849d3dac8e7d72c782c14455833796f87e6d031eb97b26"
Sep 07 21:36:48.943224 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:48.943205    2565 scope.go:117] "RemoveContainer" containerID="5df45a13b7dbd72e0e65ce46fd435c63c6cc22d53f2f4cf7f4a73b863f19219d"
Sep 07 21:36:48.949989 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:48.949944    2565 scope.go:117] "RemoveContainer" containerID="c9a0cd6fd870ae000912a66cd30b8c46036725c2f558b1baf3ca4b66b7f5b9fe"
Sep 07 21:36:51.761745 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:36:51.761711    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:37:01.664520 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:01.664481    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:37:02.990327 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:02.990265    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g" podUID="225921b1-32ce-419b-8372-c79426b60103" containerName="main" containerID="cri-o://3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f" gracePeriod=45
Sep 07 21:37:03.232159 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.232136    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:37:03.311811 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.311715    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-dshm\") pod \"225921b1-32ce-419b-8372-c79426b60103\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") "
Sep 07 21:37:03.311811 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.311775    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-tmp-dir\") pod \"225921b1-32ce-419b-8372-c79426b60103\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") "
Sep 07 21:37:03.312080 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.311896    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-model-cache\") pod \"225921b1-32ce-419b-8372-c79426b60103\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") "
Sep 07 21:37:03.312080 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rncc4\" (UniqueName: \"kubernetes.io/projected/225921b1-32ce-419b-8372-c79426b60103-kube-api-access-rncc4\") pod \"225921b1-32ce-419b-8372-c79426b60103\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") "
Sep 07 21:37:03.312080 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312049    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-home\") pod \"225921b1-32ce-419b-8372-c79426b60103\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") "
Sep 07 21:37:03.312080 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312066    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "225921b1-32ce-419b-8372-c79426b60103" (UID: "225921b1-32ce-419b-8372-c79426b60103"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:37:03.312309 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225921b1-32ce-419b-8372-c79426b60103-tls-certs\") pod \"225921b1-32ce-419b-8372-c79426b60103\" (UID: \"225921b1-32ce-419b-8372-c79426b60103\") "
Sep 07 21:37:03.312309 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312160    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-model-cache" (OuterVolumeSpecName: "model-cache") pod "225921b1-32ce-419b-8372-c79426b60103" (UID: "225921b1-32ce-419b-8372-c79426b60103"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:37:03.312513 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312308    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-home" (OuterVolumeSpecName: "home") pod "225921b1-32ce-419b-8372-c79426b60103" (UID: "225921b1-32ce-419b-8372-c79426b60103"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:37:03.312513 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312378    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:03.312513 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312391    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:03.312513 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.312401    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:03.313882 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.313856    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-dshm" (OuterVolumeSpecName: "dshm") pod "225921b1-32ce-419b-8372-c79426b60103" (UID: "225921b1-32ce-419b-8372-c79426b60103"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:37:03.314059 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.314038    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225921b1-32ce-419b-8372-c79426b60103-kube-api-access-rncc4" (OuterVolumeSpecName: "kube-api-access-rncc4") pod "225921b1-32ce-419b-8372-c79426b60103" (UID: "225921b1-32ce-419b-8372-c79426b60103"). InnerVolumeSpecName "kube-api-access-rncc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:37:03.314176 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.314161    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/225921b1-32ce-419b-8372-c79426b60103-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "225921b1-32ce-419b-8372-c79426b60103" (UID: "225921b1-32ce-419b-8372-c79426b60103"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:37:03.413033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.413003    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rncc4\" (UniqueName: \"kubernetes.io/projected/225921b1-32ce-419b-8372-c79426b60103-kube-api-access-rncc4\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:03.413033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.413026    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/225921b1-32ce-419b-8372-c79426b60103-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:03.413033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.413037    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/225921b1-32ce-419b-8372-c79426b60103-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:03.477766 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.477732    2565 generic.go:356] "Generic (PLEG): container finished" podID="225921b1-32ce-419b-8372-c79426b60103" containerID="3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f" exitCode=0
Sep 07 21:37:03.477914 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.477768    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"
Sep 07 21:37:03.477914 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.477820    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g" event={"ID":"225921b1-32ce-419b-8372-c79426b60103","Type":"ContainerDied","Data":"3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f"}
Sep 07 21:37:03.477914 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.477856    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g" event={"ID":"225921b1-32ce-419b-8372-c79426b60103","Type":"ContainerDied","Data":"123c069d0e47bf296df1e4fe8fe8ed34fe510014764c96b6f1373809309f40ae"}
Sep 07 21:37:03.477914 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.477883    2565 scope.go:117] "RemoveContainer" containerID="3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f"
Sep 07 21:37:03.486685 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.486671    2565 scope.go:117] "RemoveContainer" containerID="3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f"
Sep 07 21:37:03.486945 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:37:03.486921    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f\": container with ID starting with 3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f not found: ID does not exist" containerID="3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f"
Sep 07 21:37:03.487090 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.486954    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f"} err="failed to get container status \"3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f\": rpc error: code = NotFound desc = could not find container \"3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f\": container with ID starting with 3106f6c92d6056461b44b1043ec4ff2b07e73c874399102749d4eb453ddc4d9f not found: ID does not exist"
Sep 07 21:37:03.497749 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.497724    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"]
Sep 07 21:37:03.499885 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.499867    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8258edf4-kserve-778f5675c5-v455g"]
Sep 07 21:37:03.946160 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:03.946125    2565 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46" podUID="426cb757-74ee-4225-a58b-341a7474f8bf" containerName="main" containerID="cri-o://91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1" gracePeriod=45
Sep 07 21:37:04.189335 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.189312    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:37:04.320136 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320054    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-dshm\") pod \"426cb757-74ee-4225-a58b-341a7474f8bf\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") "
Sep 07 21:37:04.320136 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320105    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426cb757-74ee-4225-a58b-341a7474f8bf-tls-certs\") pod \"426cb757-74ee-4225-a58b-341a7474f8bf\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") "
Sep 07 21:37:04.320136 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320133    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-home\") pod \"426cb757-74ee-4225-a58b-341a7474f8bf\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") "
Sep 07 21:37:04.320432 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320149    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-tmp-dir\") pod \"426cb757-74ee-4225-a58b-341a7474f8bf\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") "
Sep 07 21:37:04.320432 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320278    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-model-cache\") pod \"426cb757-74ee-4225-a58b-341a7474f8bf\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") "
Sep 07 21:37:04.320432 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320347    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drg5b\" (UniqueName: \"kubernetes.io/projected/426cb757-74ee-4225-a58b-341a7474f8bf-kube-api-access-drg5b\") pod \"426cb757-74ee-4225-a58b-341a7474f8bf\" (UID: \"426cb757-74ee-4225-a58b-341a7474f8bf\") "
Sep 07 21:37:04.320432 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320355    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-home" (OuterVolumeSpecName: "home") pod "426cb757-74ee-4225-a58b-341a7474f8bf" (UID: "426cb757-74ee-4225-a58b-341a7474f8bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:37:04.320432 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320378    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "426cb757-74ee-4225-a58b-341a7474f8bf" (UID: "426cb757-74ee-4225-a58b-341a7474f8bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:37:04.320641 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320517    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-model-cache" (OuterVolumeSpecName: "model-cache") pod "426cb757-74ee-4225-a58b-341a7474f8bf" (UID: "426cb757-74ee-4225-a58b-341a7474f8bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:37:04.320703 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320682    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-home\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:04.320738 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320709    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-tmp-dir\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:04.320738 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.320722    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-model-cache\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:04.322338 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.322309    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-dshm" (OuterVolumeSpecName: "dshm") pod "426cb757-74ee-4225-a58b-341a7474f8bf" (UID: "426cb757-74ee-4225-a58b-341a7474f8bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 21:37:04.322441 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.322338    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/426cb757-74ee-4225-a58b-341a7474f8bf-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "426cb757-74ee-4225-a58b-341a7474f8bf" (UID: "426cb757-74ee-4225-a58b-341a7474f8bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 21:37:04.322441 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.322361    2565 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426cb757-74ee-4225-a58b-341a7474f8bf-kube-api-access-drg5b" (OuterVolumeSpecName: "kube-api-access-drg5b") pod "426cb757-74ee-4225-a58b-341a7474f8bf" (UID: "426cb757-74ee-4225-a58b-341a7474f8bf"). InnerVolumeSpecName "kube-api-access-drg5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 21:37:04.363150 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.363114    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="225921b1-32ce-419b-8372-c79426b60103" path="/var/lib/kubelet/pods/225921b1-32ce-419b-8372-c79426b60103/volumes"
Sep 07 21:37:04.421435 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.421412    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/426cb757-74ee-4225-a58b-341a7474f8bf-dshm\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:04.421435 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.421432    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/426cb757-74ee-4225-a58b-341a7474f8bf-tls-certs\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:04.421616 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.421443    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drg5b\" (UniqueName: \"kubernetes.io/projected/426cb757-74ee-4225-a58b-341a7474f8bf-kube-api-access-drg5b\") on node \"ip-10-0-128-28.ec2.internal\" DevicePath \"\""
Sep 07 21:37:04.488675 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.488640    2565 generic.go:356] "Generic (PLEG): container finished" podID="426cb757-74ee-4225-a58b-341a7474f8bf" containerID="91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1" exitCode=0
Sep 07 21:37:04.488869 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.488693    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46" event={"ID":"426cb757-74ee-4225-a58b-341a7474f8bf","Type":"ContainerDied","Data":"91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1"}
Sep 07 21:37:04.488869 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.488721    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46" event={"ID":"426cb757-74ee-4225-a58b-341a7474f8bf","Type":"ContainerDied","Data":"1e4854871c12d200b2af87db8a6bdbc47fbce5d99136fc0b73efaa36e559f6f6"}
Sep 07 21:37:04.488869 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.488736    2565 scope.go:117] "RemoveContainer" containerID="91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1"
Sep 07 21:37:04.488869 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.488691    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"
Sep 07 21:37:04.497886 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.497864    2565 scope.go:117] "RemoveContainer" containerID="91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1"
Sep 07 21:37:04.498190 ip-10-0-128-28 kubenswrapper[2565]: E0907 21:37:04.498165    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1\": container with ID starting with 91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1 not found: ID does not exist" containerID="91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1"
Sep 07 21:37:04.498316 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.498195    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1"} err="failed to get container status \"91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1\": rpc error: code = NotFound desc = could not find container \"91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1\": container with ID starting with 91446c8479d9f2426d2dbcc611158ed381c57136b21b127e49009375d79861c1 not found: ID does not exist"
Sep 07 21:37:04.505919 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.505896    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"]
Sep 07 21:37:04.509340 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:04.509308    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8258edf4-kserve-7cc9977475-2dg46"]
Sep 07 21:37:06.364748 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:06.364712    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="426cb757-74ee-4225-a58b-341a7474f8bf" path="/var/lib/kubelet/pods/426cb757-74ee-4225-a58b-341a7474f8bf/volumes"
Sep 07 21:37:08.458792 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:08.458749    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:37:15.954607 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:15.954567    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:37:25.357598 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:25.357562    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:37:33.456120 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:33.456081    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:37:39.955274 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:39.955237    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-c95tz"]
Sep 07 21:37:42.858217 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:42.858120    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9c5f48f5-p9vz7_b4d8d43e-38d5-4f75-a1eb-d238eff1193a/authorino/0.log"
Sep 07 21:37:48.616020 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:48.615985    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56489b9975-275kj_e870ee56-e294-4550-9a9b-352b31377554/copy-manifests/0.log"
Sep 07 21:37:48.622207 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:48.622184    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56489b9975-275kj_e870ee56-e294-4550-9a9b-352b31377554/manager/0.log"
Sep 07 21:37:48.976525 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:48.976446    2565 scope.go:117] "RemoveContainer" containerID="cd7e4875db9443a5ec1b1c3236c0a4c52c092d91f746b16b86ee16bd3eda6c03"
Sep 07 21:37:48.984251 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:48.984230    2565 scope.go:117] "RemoveContainer" containerID="5c9f50004a15ad8bf3b304f392b4e7e533fd9ede699e073b2c58edd5d8b9f51b"
Sep 07 21:37:49.527802 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:49.527772    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813610-jrhdn_07490ce7-59b8-404d-9d5d-f4a50083aa01/cleanup/0.log"
Sep 07 21:37:50.400662 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:50.400628    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9c5f48f5-p9vz7_b4d8d43e-38d5-4f75-a1eb-d238eff1193a/authorino/0.log"
Sep 07 21:37:50.726490 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:50.726416    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-wx74s_685d1787-efd6-4081-8e42-11d63d5e9ad1/kuadrant-console-plugin/0.log"
Sep 07 21:37:50.971488 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:50.971462    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c95tz_3383060f-16e7-4691-a7e2-02d67e321513/limitador/0.log"
Sep 07 21:37:51.750124 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:51.750097    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b8d644dc-tc66s_146296e4-520f-4615-b9f8-3127417fb4de/kube-auth-proxy/0.log"
Sep 07 21:37:52.202174 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:52.202147    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6545c9995b-hxggn_f51d6348-378e-49d1-8c95-65c42db5d93a/router/0.log"
Sep 07 21:37:52.757555 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:52.757530    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-njjbk_6be4e6b4-d4f7-4c07-9f52-701aa93836e1/main/0.log"
Sep 07 21:37:53.206712 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:53.206681    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-qvv7j_76d3922c-60fe-4726-ae00-74ffe2f164f4/main/0.log"
Sep 07 21:37:59.032726 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.032690    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z6sq6/must-gather-9ffwh"]
Sep 07 21:37:59.033148 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.033030    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="225921b1-32ce-419b-8372-c79426b60103" containerName="main"
Sep 07 21:37:59.033148 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.033042    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="225921b1-32ce-419b-8372-c79426b60103" containerName="main"
Sep 07 21:37:59.033148 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.033051    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="426cb757-74ee-4225-a58b-341a7474f8bf" containerName="main"
Sep 07 21:37:59.033148 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.033056    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="426cb757-74ee-4225-a58b-341a7474f8bf" containerName="main"
Sep 07 21:37:59.033148 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.033116    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="225921b1-32ce-419b-8372-c79426b60103" containerName="main"
Sep 07 21:37:59.033148 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.033127    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="426cb757-74ee-4225-a58b-341a7474f8bf" containerName="main"
Sep 07 21:37:59.036097 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.036076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6sq6/must-gather-9ffwh"
Sep 07 21:37:59.038466 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.038446    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6sq6\"/\"kube-root-ca.crt\""
Sep 07 21:37:59.038816 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.038795    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z6sq6\"/\"openshift-service-ca.crt\""
Sep 07 21:37:59.039417 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.039403    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z6sq6\"/\"default-dockercfg-6fhnj\""
Sep 07 21:37:59.045781 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.045757    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6sq6/must-gather-9ffwh"]
Sep 07 21:37:59.182120 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.182088    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a7e58ffd-e4e3-407b-8b82-5c573c5245af-must-gather-output\") pod \"must-gather-9ffwh\" (UID: \"a7e58ffd-e4e3-407b-8b82-5c573c5245af\") " pod="openshift-must-gather-z6sq6/must-gather-9ffwh"
Sep 07 21:37:59.182284 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.182136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zn5z\" (UniqueName: \"kubernetes.io/projected/a7e58ffd-e4e3-407b-8b82-5c573c5245af-kube-api-access-6zn5z\") pod \"must-gather-9ffwh\" (UID: \"a7e58ffd-e4e3-407b-8b82-5c573c5245af\") " pod="openshift-must-gather-z6sq6/must-gather-9ffwh"
Sep 07 21:37:59.283530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.283458    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zn5z\" (UniqueName: \"kubernetes.io/projected/a7e58ffd-e4e3-407b-8b82-5c573c5245af-kube-api-access-6zn5z\") pod \"must-gather-9ffwh\" (UID: \"a7e58ffd-e4e3-407b-8b82-5c573c5245af\") " pod="openshift-must-gather-z6sq6/must-gather-9ffwh"
Sep 07 21:37:59.283530 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.283524    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a7e58ffd-e4e3-407b-8b82-5c573c5245af-must-gather-output\") pod \"must-gather-9ffwh\" (UID: \"a7e58ffd-e4e3-407b-8b82-5c573c5245af\") " pod="openshift-must-gather-z6sq6/must-gather-9ffwh"
Sep 07 21:37:59.283832 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.283816    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a7e58ffd-e4e3-407b-8b82-5c573c5245af-must-gather-output\") pod \"must-gather-9ffwh\" (UID: \"a7e58ffd-e4e3-407b-8b82-5c573c5245af\") " pod="openshift-must-gather-z6sq6/must-gather-9ffwh"
Sep 07 21:37:59.291232 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.291203    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zn5z\" (UniqueName: \"kubernetes.io/projected/a7e58ffd-e4e3-407b-8b82-5c573c5245af-kube-api-access-6zn5z\") pod \"must-gather-9ffwh\" (UID: \"a7e58ffd-e4e3-407b-8b82-5c573c5245af\") " pod="openshift-must-gather-z6sq6/must-gather-9ffwh"
Sep 07 21:37:59.345400 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.345375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6sq6/must-gather-9ffwh"
Sep 07 21:37:59.614377 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.614346    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6sq6/must-gather-9ffwh"]
Sep 07 21:37:59.617363 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:37:59.617339    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7e58ffd_e4e3_407b_8b82_5c573c5245af.slice/crio-ed418a890e812ab0f3e2727be80717030761afe0bb35bb00e1b23de5466bdc02 WatchSource:0}: Error finding container ed418a890e812ab0f3e2727be80717030761afe0bb35bb00e1b23de5466bdc02: Status 404 returned error can't find the container with id ed418a890e812ab0f3e2727be80717030761afe0bb35bb00e1b23de5466bdc02
Sep 07 21:37:59.619550 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.619531    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 21:37:59.742923 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:37:59.742893    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6sq6/must-gather-9ffwh" event={"ID":"a7e58ffd-e4e3-407b-8b82-5c573c5245af","Type":"ContainerStarted","Data":"ed418a890e812ab0f3e2727be80717030761afe0bb35bb00e1b23de5466bdc02"}
Sep 07 21:38:00.751870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:00.751707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6sq6/must-gather-9ffwh" event={"ID":"a7e58ffd-e4e3-407b-8b82-5c573c5245af","Type":"ContainerStarted","Data":"d96c56d3af0ce993158575482e92cb54bd7f2f6afc8315d1f1a4ba94bab5ca6d"}
Sep 07 21:38:00.751870 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:00.751757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6sq6/must-gather-9ffwh" event={"ID":"a7e58ffd-e4e3-407b-8b82-5c573c5245af","Type":"ContainerStarted","Data":"eea26e0117958aacae3508a905c0029969169034de869a5bcd96f6c5b2ef16de"}
Sep 07 21:38:00.768935 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:00.768878    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z6sq6/must-gather-9ffwh" podStartSLOduration=0.982565223 podStartE2EDuration="1.768858458s" totalImagesPullingTime="786.293235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 21:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:38:00.7665137 +0000 UTC m=+2172.848041042" watchObservedRunningTime="2026-09-07 21:38:00.768858458 +0000 UTC m=+2172.850385799"
Sep 07 21:38:01.944373 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:01.944337    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9tlvn_12c4533d-f234-4b23-b343-c3cebf3279e9/global-pull-secret-syncer/0.log"
Sep 07 21:38:02.106071 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:02.106045    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-w9r8n_c541d1bb-269e-436b-95da-76cde9b0c2e8/konnectivity-agent/0.log"
Sep 07 21:38:02.127734 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:02.127705    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-28.ec2.internal_1a426b2798cb1a9b52d39c4792ff821c/haproxy/0.log"
Sep 07 21:38:06.503982 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:06.503018    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6c9c5f48f5-p9vz7_b4d8d43e-38d5-4f75-a1eb-d238eff1193a/authorino/0.log"
Sep 07 21:38:06.600747 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:06.600716    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-wx74s_685d1787-efd6-4081-8e42-11d63d5e9ad1/kuadrant-console-plugin/0.log"
Sep 07 21:38:06.702671 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:06.702644    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-c95tz_3383060f-16e7-4691-a7e2-02d67e321513/limitador/0.log"
Sep 07 21:38:08.436259 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:08.436229    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-nspqx_8ef9a728-9e2b-4341-8554-7bcabaecdf7b/kube-state-metrics/0.log"
Sep 07 21:38:08.468892 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:08.468866    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-nspqx_8ef9a728-9e2b-4341-8554-7bcabaecdf7b/kube-rbac-proxy-main/0.log"
Sep 07 21:38:08.490308 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:08.490279    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-74cb57df7-nspqx_8ef9a728-9e2b-4341-8554-7bcabaecdf7b/kube-rbac-proxy-self/0.log"
Sep 07 21:38:08.517149 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:08.517121    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-787485f48c-tnzkv_899d0185-7c8b-4477-991d-a369cc0fd262/metrics-server/0.log"
Sep 07 21:38:08.733385 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:08.733301    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxc5f_aa52f873-c713-46b2-b054-0bce67a3111a/node-exporter/0.log"
Sep 07 21:38:08.756098 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:08.756072    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxc5f_aa52f873-c713-46b2-b054-0bce67a3111a/kube-rbac-proxy/0.log"
Sep 07 21:38:08.778581 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:08.778556    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sxc5f_aa52f873-c713-46b2-b054-0bce67a3111a/init-textfile/0.log"
Sep 07 21:38:09.238863 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:09.238826    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c676d7495-4qq7m_47253a1b-63ed-4ab6-9532-83eae4f7fdbd/thanos-query/0.log"
Sep 07 21:38:09.262042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:09.261935    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c676d7495-4qq7m_47253a1b-63ed-4ab6-9532-83eae4f7fdbd/kube-rbac-proxy-web/0.log"
Sep 07 21:38:09.291150 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:09.291102    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c676d7495-4qq7m_47253a1b-63ed-4ab6-9532-83eae4f7fdbd/kube-rbac-proxy/0.log"
Sep 07 21:38:09.313328 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:09.313300    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c676d7495-4qq7m_47253a1b-63ed-4ab6-9532-83eae4f7fdbd/prom-label-proxy/0.log"
Sep 07 21:38:09.341158 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:09.341069    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c676d7495-4qq7m_47253a1b-63ed-4ab6-9532-83eae4f7fdbd/kube-rbac-proxy-rules/0.log"
Sep 07 21:38:09.371939 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:09.371904    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6c676d7495-4qq7m_47253a1b-63ed-4ab6-9532-83eae4f7fdbd/kube-rbac-proxy-metrics/0.log"
Sep 07 21:38:10.985983 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:10.985942    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-x85fw_cd8daeeb-181e-4c8b-b97b-0c06a98faaee/networking-console-plugin/0.log"
Sep 07 21:38:12.112039 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.111956    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"]
Sep 07 21:38:12.114952 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.114930    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.126587 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.126562    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"]
Sep 07 21:38:12.213145 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.213105    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-podres\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.213453 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.213430    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-proc\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.213656 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.213617    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-lib-modules\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.213832 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.213814    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7nd2\" (UniqueName: \"kubernetes.io/projected/dfea9297-2b3d-4047-a7cc-d87e2220f402-kube-api-access-n7nd2\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.214033 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.214016    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-sys\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.315329 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.315283    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7nd2\" (UniqueName: \"kubernetes.io/projected/dfea9297-2b3d-4047-a7cc-d87e2220f402-kube-api-access-n7nd2\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.315532 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.315366    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-sys\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.315532 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.315401    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-podres\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.315532 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.315432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-proc\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.315532 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.315467    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-lib-modules\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.315757 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.315639    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-lib-modules\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.316079 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.316050    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-sys\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.316223 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.316178    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-podres\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.316291 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.316233    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dfea9297-2b3d-4047-a7cc-d87e2220f402-proc\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.325507 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.325454    2565 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7nd2\" (UniqueName: \"kubernetes.io/projected/dfea9297-2b3d-4047-a7cc-d87e2220f402-kube-api-access-n7nd2\") pod \"perf-node-gather-daemonset-9m97z\" (UID: \"dfea9297-2b3d-4047-a7cc-d87e2220f402\") " pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.431356 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.431262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:12.718219 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.717829    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"]
Sep 07 21:38:12.723694 ip-10-0-128-28 kubenswrapper[2565]: W0907 21:38:12.723646    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poddfea9297_2b3d_4047_a7cc_d87e2220f402.slice/crio-30da448c4929428de6adb56a2e0c620c14bdc416e5a6090f920d7f6b9868c67e WatchSource:0}: Error finding container 30da448c4929428de6adb56a2e0c620c14bdc416e5a6090f920d7f6b9868c67e: Status 404 returned error can't find the container with id 30da448c4929428de6adb56a2e0c620c14bdc416e5a6090f920d7f6b9868c67e
Sep 07 21:38:12.828245 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:12.828200    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z" event={"ID":"dfea9297-2b3d-4047-a7cc-d87e2220f402","Type":"ContainerStarted","Data":"30da448c4929428de6adb56a2e0c620c14bdc416e5a6090f920d7f6b9868c67e"}
Sep 07 21:38:13.836104 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:13.836043    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z" event={"ID":"dfea9297-2b3d-4047-a7cc-d87e2220f402","Type":"ContainerStarted","Data":"1203cf43f89d7134cc82bfb104f9e86bf192cacc74d8ccfee121da559492b8eb"}
Sep 07 21:38:13.836594 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:13.836173    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:13.855266 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:13.855212    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z" podStartSLOduration=1.855195879 podStartE2EDuration="1.855195879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 21:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 21:38:13.852671123 +0000 UTC m=+2185.934198466" watchObservedRunningTime="2026-09-07 21:38:13.855195879 +0000 UTC m=+2185.936723221"
Sep 07 21:38:14.134613 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:14.134537    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z6twk_917948a6-64e5-49c9-9bc9-31f75632da0b/dns/0.log"
Sep 07 21:38:14.156457 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:14.156430    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z6twk_917948a6-64e5-49c9-9bc9-31f75632da0b/kube-rbac-proxy/0.log"
Sep 07 21:38:14.182417 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:14.182388    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-22b4b_57134fda-ea48-44a8-920e-b435bb868eb4/dns-node-resolver/0.log"
Sep 07 21:38:14.815639 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:14.815610    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h94gv_02eff29a-6dd4-495c-84b1-97c952ad2b0b/node-ca/0.log"
Sep 07 21:38:16.091154 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:16.091125    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b8d644dc-tc66s_146296e4-520f-4615-b9f8-3127417fb4de/kube-auth-proxy/0.log"
Sep 07 21:38:16.219458 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:16.219427    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6545c9995b-hxggn_f51d6348-378e-49d1-8c95-65c42db5d93a/router/0.log"
Sep 07 21:38:17.074116 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:17.074089    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ggj9j_54304c6d-a18b-46f3-929b-58f0acfe7751/serve-healthcheck-canary/0.log"
Sep 07 21:38:17.572157 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:17.572128    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-qb7kv_c9b1579c-00e4-4922-8bd5-87f58f77c496/insights-operator/0.log"
Sep 07 21:38:17.572615 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:17.572388    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-qb7kv_c9b1579c-00e4-4922-8bd5-87f58f77c496/insights-operator/1.log"
Sep 07 21:38:17.591800 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:17.591777    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fq5qz_f9250458-8d8f-42c7-bc7e-0dfca2791158/kube-rbac-proxy/0.log"
Sep 07 21:38:17.612787 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:17.612766    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fq5qz_f9250458-8d8f-42c7-bc7e-0dfca2791158/exporter/0.log"
Sep 07 21:38:17.638441 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:17.638417    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fq5qz_f9250458-8d8f-42c7-bc7e-0dfca2791158/extractor/0.log"
Sep 07 21:38:19.850388 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:19.850360    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z6sq6/perf-node-gather-daemonset-9m97z"
Sep 07 21:38:20.183042 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:20.182946    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56489b9975-275kj_e870ee56-e294-4550-9a9b-352b31377554/manager/0.log"
Sep 07 21:38:20.207401 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:20.207373    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-56489b9975-275kj_e870ee56-e294-4550-9a9b-352b31377554/copy-manifests/0.log"
Sep 07 21:38:21.775031 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:21.774990    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-75546db857-h8g6p_75ebc97d-4abc-4a7e-bbea-a509781a7975/manager/0.log"
Sep 07 21:38:22.834664 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:22.834628    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b8d644dc-tc66s_146296e4-520f-4615-b9f8-3127417fb4de/kube-auth-proxy/0.log"
Sep 07 21:38:22.923630 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:22.923579    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6545c9995b-hxggn_f51d6348-378e-49d1-8c95-65c42db5d93a/router/0.log"
Sep 07 21:38:24.809236 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:24.809202    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58b8d644dc-tc66s_146296e4-520f-4615-b9f8-3127417fb4de/kube-auth-proxy/0.log"
Sep 07 21:38:24.913434 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:24.913397    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6545c9995b-hxggn_f51d6348-378e-49d1-8c95-65c42db5d93a/router/0.log"
Sep 07 21:38:28.373884 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:28.373857    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hds9_04e701e8-8302-4722-865f-c23db8b82045/kube-multus-additional-cni-plugins/0.log"
Sep 07 21:38:28.394571 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:28.394548    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hds9_04e701e8-8302-4722-865f-c23db8b82045/egress-router-binary-copy/0.log"
Sep 07 21:38:28.416429 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:28.416407    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hds9_04e701e8-8302-4722-865f-c23db8b82045/cni-plugins/0.log"
Sep 07 21:38:28.437495 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:28.437472    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hds9_04e701e8-8302-4722-865f-c23db8b82045/bond-cni-plugin/0.log"
Sep 07 21:38:28.457622 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:28.457600    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hds9_04e701e8-8302-4722-865f-c23db8b82045/routeoverride-cni/0.log"
Sep 07 21:38:28.477528 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:28.477510    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hds9_04e701e8-8302-4722-865f-c23db8b82045/whereabouts-cni-bincopy/0.log"
Sep 07 21:38:28.497822 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:28.497797    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-6hds9_04e701e8-8302-4722-865f-c23db8b82045/whereabouts-cni/0.log"
Sep 07 21:38:28.856561 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:28.856521    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dpq7j_883e3628-8e19-4c1a-933c-d7209b86b4cd/kube-multus/0.log"
Sep 07 21:38:29.001327 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:29.001244    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xbp8h_2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8/network-metrics-daemon/0.log"
Sep 07 21:38:29.021016 ip-10-0-128-28 kubenswrapper[2565]: I0907 21:38:29.020990    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-xbp8h_2499acf3-9ad9-4ddd-9c3d-c7bedecd1ac8/kube-rbac-proxy/0.log"